Agents
Building computer-use agents from scratch, inspired by Meta's Muse
November 1, 2026
Sarthak Gupta
Software & ML Engineer
Summary
A while ago Meta posted a Research Engineer role for computer-use agents on Muse. While Meta, Microsoft, and frontier labs are already building in this space, I have a feeling that the majority of tech companies will start implementing some form of CUA technology, similarly how companies followed suit with RAG. Hence, I want to understand the work well enough that I can do it, so I started building the pieces myself in a public repo, cua-learning. There is no end-to-end course on this, so I track the work in GitHub issues and pull requests, and I use GPT models and Claude Code as reviewers that play the role of a senior engineer.
I'm simulating decisions a Research Engineer at Meta Superintelligence Labs would make on this kind of project. On a project like this, that person spends most of their time on the infrastructure that makes training and evaluating the model trustworthy: every rollout starts from a known state, every failure can be traced to the model or to the infrastructure, and the reward is hard to fake. Those three properties decided most of what follows.
I am updating this page live as I work on this engineering project. Last updated November 14, 2026.
The recording below shows the agent as of November 13. It's an early first version running one deliberately small task: Claude opens LibreOffice Writer, types one sentence, and saves the file. It isn't benchmarked or tuned. When this was recorded the evaluator didn't check agent runs yet. Since November 14 it does, and the two runs it has scored so far both passed. Most of the work has gone into the infrastructure around the model, which is what the rest of this post covers.
Progress snapshot at November 13, 2026 showing v1 of the agent loop.
- Where it stands: A model chooses the actions, and a separate check on the Mac decides whether the run passed. One command restores a clean snapshot, boots the VM, runs the agent loop, copies the run off the VM, evaluates the saved document on the Mac, writes
result.json, and shuts the VM down. Two agent runs have been scored so far, and both passed. The scripted version still exists as the reference. - How a run is scored: The model's own
donecan't make a run pass. A run passes only if the evaluator, running on the Mac against a copy of the document, finds exactly the expected sentence, and the agent also said it was done within 12 steps. - Why it took three weeks to get there: In a computer-use system the model is one component. It also needs a desktop it can reset, a way to send keystrokes and read the screen, and a check that decides whether the task was done. I wanted those to work and to fail clearly before adding a model whose mistakes would be mixed in with mine.
- What I built so far: An executor, an evaluator, an orchestrator that writes a machine-readable result and exit code for every run, a host controller that runs the whole thing remotely, a one-command clean run (restore snapshot, boot, wait until ready, run, copy results to the Mac), the agent loop, and host-side scoring for the agent's runs.
- What I tested: Two deliberate faults, a wrong sentence and a broken GUI command. The second one exposed a bug in my own orchestrator. Later, an agent run that ended in an error turned out to be a typo in my click bounds check.
- What a code review changed: A Claude bot review of the host controller found that a run which hit the time limit would keep typing into the VM after the Mac had given up on it. Fixing that led to two layers of timeouts with distinct exit codes, and a backlog for the findings I chose not to fix yet.
Build log
Every step of this project is a GitHub issue, a pull request, or both. Open any entry for a short summary, links to the issue and PR, and the section of this post that covers it.
Scoring agent runs
Nov 14Clean-snapshot agent runsNov 14 · Commit 62222ec
python3 agent_probe.py --clean-agent-run restores the baseline snapshot, waits for SSH and X11, runs the agent, scores the run on the Mac, and shuts the VM down. Its first committed run took eight steps and passed.
Nov 14Independent evaluation of model-driven runsNov 14 · Issue 23 · PR 24
After the agent loop ends, the run directory is copied to the Mac and the original evaluator checks document.odt there. result.json records the outcome, and a run passes only if the evaluator passes it and the agent returned done.
Model in the loop
Nov 13Introduce model-driven computer useNov 13 · Issue 21 · PR 22
agent_probe.py asks Claude for one structured action per screenshot: type, keypress, click, or done. Python checks it against a schema and allowlists, xdotool runs it, and every step is saved to trajectory.json. The first full run opened Writer, typed the sentence, and saved the file in five actions.
Reproducible episodes
Nov 12One-command reproducible experimentsNov 12 · Issue 19 · PR 20
python3 host_controller.py --run-from-clean restores the baseline snapshot, boots the VM, waits for SSH and X11, runs the experiment, collects the artifacts, and exits with the experiment's exit code.
Nov 12VM readiness checksNov 12 · Issue 17 · PR 18
wait_for_ssh() and wait_for_gui() poll every 2 seconds until the guest answers as the right user and the X11 display reports a valid size, within 90 and 120 seconds. They replace one-time checks that failed on a fresh boot.
Nov 11Reproducible VM snapshots and resetsNov 11 · Issue 15 · PR 16
reset_vm() restores the disk-only cua-gui-ssh-baseline snapshot and starts the VM, so every episode gets a real boot with no processes or application state left over from earlier runs.
Nov 11Collect experiment artifacts on the hostNov 11 · Issue 13 · PR 14
The guest prints its run ID, and the host copies that run directory to collected_runs/ on the Mac with scp, so results survive the next snapshot restore.
Host control and code review
Nov 7Automate VM lifecycle and host-to-guest executionNov 7 · Issue 6 · PR 10
host_controller.py runs the harness in the guest over SSH with a 170-second guest timeout, a 200-second host timeout, and a separate exit code for each way a run can end. A nine-comment Claude review shaped most of it.
Nov 7BacklogNov 7 · Issue 12 · Open
Review findings I deferred on purpose: the hardcoded guest IP, GUI paths that assume UID 1000, and reusing one SSH connection per run.
Nov 7Claude Code workflow for @claude mentionsNov 7 · PR 11
A GitHub Actions workflow that runs Claude Code when @claude is mentioned in an issue or pull request, set up for senior-engineer reviews on PRs.
Nov 1Migrate to SSH for executing commandsNov 1 · Issue 9
utmctl exec ran commands as root and sometimes returned empty output. Commands moved to SSH with a dedicated key, and utmctl now handles only the VM lifecycle.
Oct 28Execute commands as a non-root Ubuntu userOct 28 · Issue 8
The QEMU guest agent ran everything as root. Commands now run as the regular desktop user, which is safer for an agent and is the user that owns the display.
Oct 28Scaffold v0 host_controller.pyOct 28 · Issue 7
The first controller on the Mac: confirm the VM is running, then run uname and id in the guest.
Oct 28Set up utmctl and the QEMU guest agentOct 28 · Issue 5
Explored UTM's command-line tool and installed qemu-guest-agent so the Mac could reach the guest.
Evaluation harness
Oct 28v1 end-to-end evaluation harnessOct 28 · Issue 1 · PR 4
run_task.py joins the executor and evaluator into one repeatable experiment with its own run directory, a results.json, and a nonzero exit code on failure.
Oct 28Fault injectionOct 28 · Issue 2
Injected a wrong sentence and a broken executor command, one at a time. The harness classified both correctly, and the second one exposed a bug in the orchestrator's finally block.
Oct 27AI senior-engineer reviewOct 27 · Issue 3
ChatGPT reviewed the harness before fault injection and found three problems: exit code 0 on failure, a global event log, and a wrong type annotation.
Oct 22First commitOct 22 · Commit 89a8504
executor.py drives LibreOffice Writer with xdotool: open it, type one sentence, and save the file through the GUI.
Why the harness comes first
A computer-use agent looks at a screen, decides on an action, performs it, and looks again. Most demos show that loop. Training or evaluating one at scale needs a lot around it:
- An environment that is a real desktop with real applications, isolated from anything that matters, and restorable to a known state between runs.
- An action layer that turns "type this" or "press Ctrl+S" into input events the desktop accepts.
- Observations, mostly screenshots, plus logs for whoever has to debug a failed run later.
- A verifier that looks at the end state and says whether the task was done. For evaluation this is the score. For reinforcement learning it is the reward.
- An orchestrator that runs many of these, records what happened, and reports results in a form another program can read.
Public benchmarks are built the same way. OSWorld, for example, runs tasks in Ubuntu VMs, includes LibreOffice tasks, resets from snapshots, and scores each task with a script that inspects the resulting files or system state. I didn't copy its design, but I found that I had ended up with the same parts for the same reasons.
I deliberately made the policy a fixed script. If a model chose the actions, every failure would raise the question of whether the model did something wrong or the harness did. With a script, the actions are the same every run, so anything that breaks is the environment, the action layer, or the evaluator. On a training team, that distinction decides whose problem a failed rollout is, and a harness that can't make it wastes researchers' time on bugs that were never in the model! The model went in only after the environment, the reset, and the evaluator had been tested, which is the last part of this post.
The first task
The task is small: open LibreOffice Writer, type Hello from my Python computer-use executor!, save it as an .odt file through the Save As dialog, and check the saved file. I saved through the GUI on purpose, because writing the file from Python would have been one line and would have tested nothing about the desktop.
Even this task touches most of the problems a real agent hits: an application that takes a variable time to start, windows that need focus, keyboard shortcuts, a modal dialog, and a file that may or may not appear on disk.
Setting up the environment
I run everything on an Apple Silicon Mac, so the guest is Ubuntu 24.04.5 LTS for ARM64 under UTM, which uses QEMU. Before writing any automation I took a clean snapshot, modified a test file, restored the snapshot, and confirmed the change was gone. Resetting to a known state is the main reason to use a VM, and I wanted to see it work before relying on it.
The first real problem was the display server. Ubuntu 24.04 defaults to Wayland, and xdotool, the tool I use to send keystrokes and find windows, works through X11. I spent time on a Wayland/Xorg compatibility issue before switching the GNOME session to X11. After that, xdotool could control the keyboard and mouse and scrot could take screenshots.
echo "$XDG_SESSION_TYPE" # expected: x11
xdotool getdisplaygeometry
The executor
executor.py is the hand-written policy. It does what a model would eventually do, using the same primitives:
- Launch
libreoffice --writerin the background. - Poll
xdotool search --onlyvisible --name "LibreOffice Writer"once a second for up to 20 seconds, and take the last window ID it returns as the newest window. - Activate the window with
windowactivate --sync, press Ctrl+N for a new document, and takebefore.png. - Type the sentence with a 40 ms delay between keystrokes and take
after_typing.png. - Press Ctrl+Shift+S for Save As, Ctrl+L to get a path field, type the output path, and press Return.
- Poll for the file every half second for up to 10 seconds until it exists and is non-empty, then take
after_save.png.
Step 2 comes back later in this post. Taking the last matching window works when there is exactly one Writer instance, and it picks the wrong window when an earlier run left one open.
Every command goes through one helper that logs it before running it:
def execute(*args):
record("command", " ".join(args))
result = subprocess.run(
args,
capture_output=True,
text=True,
timeout=30,
check=True,
)
return result.stdout.strip()
I originally ran the command and then logged it, so a command that failed never appeared in events.json, and the log ended one step before the problem. Logging first means the last event in a failed run is the command that failed.
If anything in the executor raises, it records an error event, writes events.json in a finally block, and raises an ExecutorError with from error so the original exception (usually a subprocess.CalledProcessError) stays attached to the traceback.
The evaluator
The executor finishing without an exception tells me the keystrokes were sent. It says nothing about whether the document contains the right text. Writer could have been slow and dropped characters, focus could have moved to another window halfway through typing, or the dialog could have saved somewhere else. For that reason the evaluator ignores the executor's report and reads the saved file.
An .odt file is a ZIP archive, and the document body lives in content.xml. The evaluator uses only the standard library:
- Check that the file exists.
- Check that it is a valid ZIP and that
testzip()finds no corrupted members. - Parse
content.xmland collect the text of every paragraph in the document body. - Pass only if the paragraphs equal exactly
[expected].
My first version joined all the text in the XML into one string and checked whether the sentence appeared in it. That would pass a document with the right sentence plus three paragraphs of garbage, and it would also pick up text from parts of the XML that aren't the document body. Requiring exactly one paragraph with exactly the expected text is stricter, and for this task it is the correct definition of success. Once this check becomes a reward, a model trained against the loose version would learn to satisfy the loose version, so the strict check is the one worth having from the start.
This separation is the part I expect to carry over unchanged when a model replaces the script. The evaluator only knows the task's expected outcome and where the artifact is. It has no idea how the artifact was produced.
Turning a script into an experiment
run_task.py ties the two together. Each run gets an ID like 20261027-153012-a1b2c3 and its own directory under runs/, and the document path is inside that directory. A document left over from an earlier successful run can't satisfy a later evaluation, because the later run looks in a directory that didn't exist until it started.
Each run tracks a result with explicit fields and a status:
class ResultDict(TypedDict):
run_id: str
task_name: str
executor_completed: bool
evaluation_completed: bool
evaluation_success: bool
end_to_end_success: bool
error_type: NotRequired[str]
error_message: NotRequired[str]
status: Literal["not_started", "started_execution", "execution_completed", ...,
"executor_failed", "evaluation_failed", "harness_failed", "completed"]
Executor errors, evaluator errors, and anything else are caught separately, so a broken GUI command, a corrupt document, and a bug in the harness get different statuses. The result is written to the run directory in a finally block, and the process exits with the outcome:
return 0 if result["end_to_end_success"] else 1
if __name__ == "__main__":
raise SystemExit(main())
The exit code is easy to skip when you are running one script by hand and reading the output, but another program running the script has nothing else to go on. If a scheduler launches a thousand runs and a crashed run exits with 0, the scheduler counts it as a success. Catching an exception and printing it does not make a process fail, so the orchestrator has to report failure on purpose.
Using AI as a senior engineer
Before injecting any faults, I asked ChatGPT to review the four files the way a senior engineer would review a junior's code before an experiment. I checked each claim myself before acting on it (issue #3). It found three problems.
The orchestrator exited with 0 on failure. My run_task.py at the time caught exceptions, printed them, and finished normally. That was the problem in the previous section, and it was the most important one to fix, because a fault-injection test that can't fail tells you nothing.
The event log was global. events = EventManager() was created once when executor.py was imported. Each python3 run_task.py starts a fresh process, so it didn't cause problems yet. If I ran several tasks in one process, the third run's events.json would contain the events from the first two. That has to move into execute_task() before I run tasks in a loop or in parallel, and it is still on my list.
The evaluator's type annotation was wrong. evaluate() declared document_path: str and then called .is_file() on it, which a string doesn't have. It worked only because the caller happened to pass a Path. I fixed the annotation and moved the first file check inside the try block so path errors are handled the same way as everything else.
It also suggested the from error chaining and the order of the next steps: fix the orchestrator, confirm the happy path still passes, inject one fault, revert it, then inject the other. I followed that order.
Breaking it on purpose
Issue #2 defined two faults and what the harness should do for each. The goal was to check that the harness could tell three cases apart: the task was done correctly, the executor worked but produced the wrong document, and the executor itself failed.
Fault 1: the wrong sentence. I changed the text the executor types and left the evaluator's expected text alone. The executor ran normally and saved a valid .odt. The evaluator found the mismatch, evaluation_success was false, and echo $? printed 1. The harness behaved as intended.
Fault 2: a broken command. I introduced a failure inside the executor so it would raise an ExecutorError. The orchestrator caught it and recorded executor_failed, but working through this case showed a problem in its own finally block.
That block also wrote the evaluator's output to evaluation_log.json. When the executor fails, the evaluator never runs, so there was no evaluator output to write and the variable holding it was never assigned. The fault I injected into the executor exposed a bug in the orchestrator, which is the code that was supposed to be reporting the fault. I fixed it by skipping the evaluation log when the status is executor_failed, re-ran the test, and got the expected result. Then I reverted the injected fault and re-ran the original task to confirm it still passed with a fresh run directory.
I don't think I would have found that bug by running the happy path any number of times. It only shows up when the first stage fails, and a large evaluation run would hit that case thousands of times.
Moving control to the host
Up to this point I ran everything from a terminal inside the VM. That doesn't scale to anything interesting: restoring a snapshot from inside the VM you are restoring doesn't work, and an agent's evaluator should eventually run somewhere the agent can't reach. So the next goal (issue #6) was to drive experiments from the Mac.
First attempt: the QEMU guest agent
UTM ships a command-line tool, utmctl, that can report a VM's status, start and stop it, manage snapshots, and run commands in the guest if the QEMU guest agent is installed there. I installed qemu-guest-agent in Ubuntu, confirmed the org.qemu.guest_agent.0 virtio port existed, and wrote a first host_controller.py that did three things: check that the VM was running, run uname -s in the guest, and run id -un to see which user the commands ran as.
uname returned Linux and id -un returned root. The guest agent runs commands as root by default, which is wrong for this work in two ways. An agent shouldn't have root on the machine it is operating, and root isn't the user who owns the desktop session, so GUI commands wouldn't find the display anyway. I wrapped commands in runuser -u sarthak -- so they ran as my regular account (issue #8).
One more thing came up along the way: host_controller.py failed when run from the VS Code integrated terminal and worked from the regular macOS Terminal. As far as I can tell this is macOS protecting UTM from being controlled by other apps, since a malicious repository opened in an editor shouldn't be able to run commands against your VMs.
Switching to SSH
The guest agent turned out to be unreliable for running commands. Some commands intermittently returned empty output, and the workarounds I tried didn't fix it. A harness that sometimes gets no output can't tell a broken task from a broken channel, which is the distinction the whole project is built around.
So I split the responsibilities (issue #9). utmctl now only handles the VM lifecycle: status, start, stop, and snapshots. Commands go over SSH, which logs in as my regular user and reports the remote command's exit code, stdout, and stderr separately.
def execute_in_vm(*command, gui=False, timeout=30):
remote_command = [*(GUI_ENV if gui else []), *command]
return subprocess.run(
[
"ssh",
"-i", str(SSH_KEY),
"-o", "IdentitiesOnly=yes",
"-o", "BatchMode=yes",
"-o", "ConnectTimeout=5",
SSH_TARGET,
shlex.join(remote_command),
],
capture_output=True,
text=True,
timeout=timeout,
)
The SSH options are all there to make failures fast and visible. The connection uses a dedicated key, BatchMode makes it fail instead of waiting for a password prompt nobody will answer, and the five-second connect timeout stops a dead VM from hanging the controller. The function returns the whole CompletedProcess so the caller can check the exit code instead of trusting whatever text came back.
An SSH session isn't the desktop session, so GUI commands don't know which display to draw on or which X authority file to use. Commands that need the desktop are prefixed with the variables that point at the logged-in session: DISPLAY=:0, XAUTHORITY=/run/user/1000/gdm/Xauthority, XDG_RUNTIME_DIR=/run/user/1000, and the D-Bus session address.
Running the experiment from the Mac
With SSH working, the controller runs a short series of preflight checks and then the experiment itself (PR #10):
utmctl statusmust reportstarted, or the controller stops.uname -sover SSH must succeed. This confirms the connection, the key, and the user.id -unreports which user the commands run as.xdotool getdisplaygeometrywith the GUI environment must return a resolution. This confirms the command can reach the desktop.- Run
python3 -u run_task.pyin the guest, with the GUI environment and a time limit.
Each check fails with its own message, so a run that never started because SSH was down is reported differently from a run that started and failed. The -u flag turns off Python's output buffering so the guest's logs reach the SSH pipe as they are printed.
Using Claude as a "Senior Engineer" that lives in the repo
For this PR, I wanted the senior-engineer review to happen where a real one would, as inline comments on the pull request. PR #11 added a GitHub Actions workflow that runs Claude Code whenever someone mentions @claude in an issue or PR. The workflow's first runs on PR #10 failed within two seconds each, so I ran the same review from Claude Code on my machine and had it post its findings as inline comments on the PR, under a claude[bot] header.
I asked for a review as a senior engineer, focused on the SSH execution and VM lifecycle code, error handling, security, and maintainability. It left nine comments. The most important one was about the time limit on the experiment.
The run that keeps typing
My first version passed timeout=180 to subprocess.run for the experiment. The review pointed out two problems with that. Nothing caught the subprocess.TimeoutExpired it raises, so the controller would crash with a traceback. And killing the local ssh process doesn't stop the remote command, because no terminal is allocated for it. The guest's run_task.py would keep running after the Mac had given up on it, still typing into Writer, and LibreOffice would stay open.
The leftover window matters because of step 2 of the executor. The next run would search for Writer windows, find two, and take the last one, which might be the one the dead run was still typing into. Two runs would share one desktop, and the second run's result would depend on what the first one left behind. In RL terms, one episode's state would leak into the next, and any reward computed from that episode would be measuring a mix of both.
I split the fix into two time limits, because the guest and the host fail in different ways:
experiment = execute_in_vm(
"/usr/bin/timeout",
"--signal=TERM", # ask the run to stop at the limit
"--kill-after=5s", # force it 5 seconds later if it hasn't
"170s",
"/usr/bin/python3", "-u", GUEST_RUNNER,
gui=True,
timeout=200, # host-side limit on the ssh process
)
The guest limit is enforced inside Ubuntu by the GNU timeout utility, so it stops the run even if the SSH connection is gone. The host limit is longer and only fires when the connection itself hangs. If it does fire, the controller catches TimeoutExpired, prints whatever stdout and stderr it captured before the timeout, says that it can't confirm the remote run was stopped, and returns 124. The captured output can arrive as bytes even with text=True, so a small _to_text() helper decodes it. That helper also came from the review, which noticed the decoding was written out twice.
My first version of the exit-code check only handled 124, the code timeout returns when the run stops after SIGTERM. The follow-up review pointed out that when --kill-after has to send SIGKILL, the exit code is 128 + 9 = 137. I checked both against the GNU coreutils documentation and added the case. The controller now reports three outcomes separately: 124 for a run that stopped at its time limit, 137 for one that had to be killed, and 255 for an SSH failure where the state of the remote run is unknown.
What I fixed, deferred, and declined
I didn't apply all nine comments. I replied on the PR to say what happened to each of the main ones, and the ones worth doing later went into a backlog issue (issue #12) with a link back to the comment.
- Fixed: the uncaught timeout and the remote run it left behind, the SIGKILL exit code, an SSH check whose exit code was never read (it printed the whole
CompletedProcessobject as the guest OS),utmctlerrors that lost their stderr message, the duplicated decoding, and an outdated comment. - Backlogged: the guest IP is hardcoded even though UTM assigns it by DHCP, so it can change after a snapshot restore; the GUI paths assume UID 1000 and an Xorg session; and every call opens a new SSH connection, which connection multiplexing would avoid.
- Declined: removing
scratch.py. It's my prototyping file, and I keep it in the history on purpose as a record of how I learned things. - Deferred: the second half of the timeout comment. When the guest's
timeoutsends SIGTERM, Python exits without runningfinallyblocks, so a run that times out never writesresults.jsonorevents.json, even though a timed-out run is the one I'd most want a record of. The suggested fix is a SIGTERM handler that raisesSystemExit, plus guarding the evaluation log against runs that stop before evaluation. I left it for later because the harness is still small and I'd rather add that handling when I build the rest of the reset and cleanup path.
The review also noted that, despite the PR's title, nothing automates the VM lifecycle yet. The controller checked that the VM was running and stopped if it wasn't. The next four pull requests fixed that.
Making every episode start from the same desktop
A training run treats each attempt at a task as an episode, and the episodes are only comparable if each one starts from the same state. My runs didn't. Each one started from whatever the previous run, or I, had left on the desktop. Over two days I closed that gap in four small PRs, each with its own issue: copy results to the host, restore a snapshot, wait until the guest is ready, and tie it all into one command.
Getting the results out first
Restoring a snapshot rolls back the guest's whole disk, including runs/. If I restored before copying anything out, every screenshot, events.json and results.json from the previous run would be gone. So the first PR (#14) copies each run's directory to the Mac before anything else can touch it.
The host needs the run ID to know which directory to copy, and only the guest knows it. run_task.py now prints one line as soon as it creates the directory:
print(f"CUA_RUN_ID={RUN_ID}", flush=True)
The controller finds that line in the SSH output with an anchored regular expression and raises an error if it's missing, then copies the directory with scp -r into collected_runs/ on the Mac, using the same key and fail-fast options as every SSH call. This runs whenever the guest reports an ID, so failed runs get collected too, and those are the ones I most need to look at. It's also the first step toward evaluating outside the guest, since the artifact now exists somewhere the agent can't edit.
Restoring a clean snapshot
The second PR (#16) added reset_vm(). It restores a snapshot named cua-gui-ssh-baseline with utmctl snapshot restore and starts the VM. It refuses to run unless the VM is stopped, and the error says what state it found instead.
The snapshot is disk-only. UTM can also snapshot memory, which would resume the VM exactly where it was, with the desktop already logged in. I worked through the choice in the issue (#15) and went with disk-only because every episode then gets a real boot. Services start fresh, and no process or application state from before the snapshot comes back. The cost is that a freshly booted VM isn't ready immediately, so the harness has to check for itself when it is.
Waiting until the guest is ready
A VM that UTM reports as started is still booting. For a while SSH refuses connections, and after that the desktop session takes longer still to accept windows. My preflight checks from the previous section ran once each, so on a fresh boot they would have failed every time.
The third PR (#18) replaced them with two polling functions:
wait_for_ssh()runsid -unover SSH every 2 seconds, with 8 seconds per attempt and 90 seconds in total. It passes only when the command succeeds and the answer issarthak, so it also confirms the commands are running as the right user.wait_for_gui()runsxdotool getdisplaygeometrythrough the GUI environment on the same schedule, with 120 seconds in total. It passes only when the output is two positive integers. A command that exits 0 but prints nothing doesn't count.
When either one runs out of time, the error includes the last failure it saw, so a VM that never got SSH is reported differently from one whose desktop never came up.
One command
The last PR (#20) added --run-from-clean, which calls reset_vm() and then runs the experiment, and passes the experiment's exit code through as the process's own. A full episode is now one command:
python3 host_controller.py --run-from-clean
A batch of these would still have to run one after another on one VM, and reset_vm() expects me to have shut the VM down after the previous run. Powering off after each run is a todo in the code. Within those limits, every episode now starts from the same disk, waits for the same readiness signals, and leaves its record on the host, which is what makes the runs comparable.
Putting a model in the loop
With episodes reproducible, the next issue (#21) replaced the hand-written policy with a model. I built it in four steps over one day, and each step was a working program before I moved to the next.
Seeing what the model sees
The first version of agent_probe.py didn't act at all. It took a screenshot of the VM's desktop, copied it to the Mac, and asked Claude to describe what it saw and suggest one next action. Claude runs through the Claude Code CLI with claude -p in non-interactive mode, and it can use exactly one tool, Read, so it can open the screenshot file and nothing else. MCP tools are disallowed and each call gets at most three turns.
The --live mode wires this into the harness: check that the VM is running, wait for SSH and X11, run scrot in the guest, scp the image to an observations/ folder on the Mac, and delete the temporary copy in the guest. Looking at these descriptions first was a cheap way to check that the model could read a 1008×600 GNOME desktop before giving it any control.
A structured action space
Next, Claude had to return an action instead of a paragraph. I defined four:
type: type text into the focused fieldkeypress: press a key or shortcutclick: click a screen coordinatedone: the model believes the task is complete
The action has to match a JSON schema, which goes to the CLI through --json-schema:
ACTION_SCHEMA = {
"type": "object",
"properties": {
"action": {"type": "string", "enum": ["type", "keypress", "click", "done"]},
"text": {"type": "string"},
"key": {"type": "string"},
"x": {"type": "integer"},
"y": {"type": "integer"},
},
"required": ["action", "text", "key", "x", "y"],
"additionalProperties": False,
}
Every field is required, and unused ones come back as "" or 0. That keeps parsing simple, because every response has the same shape. The prompt also gives the screen's width and height and says where (0, 0) is, because I noticed while writing it that the model can't give useful click coordinates without knowing the coordinate system.
The schema only says what an action looks like. validate_action() decides which actions are allowed. It rejects unexpected fields and wrong types, and it limits keypress to seven keys and shortcuts: Return, Escape, Tab, Ctrl+N, Ctrl+S, Ctrl+Shift+S and Ctrl+L. In this first version, type may only enter the expected sentence or the target file path, and anything else raises an error before it reaches the VM. That allowlist is deliberately narrow for one task. It will have to become a real policy before the agent can do more than one thing.
The loop
The last step (PR #22) put this in a loop. --agent-run creates a run ID and a directory in the guest, then repeats: take a screenshot, ask Claude for one action, validate it, execute it with xdotool over SSH, wait one second, and do it again. The loop ends when Claude returns done or after 12 steps. Every step's screenshot and action are saved to a trajectory.json on the Mac in a finally block, so a run that crashes still leaves its record.
I wrote down the trade-offs in the issue as I made them:
- One action per model call. Planning the whole sequence up front would be faster and cheaper, but GUI interactions don't always do what you expect, so the model should see what actually happened before it decides the next step.
- Claude decides, Python executes. Claude proposes an action, Python validates it, and
xdotoolruns it. The model never gets a shell, which also keeps it away from the evaluator's files. - History as text. Each
claude -pcall starts with no memory, so the loop passes back the list of earlier actions. That's fine for a task this short. For longer tasks the context will grow, and that will need a different approach. - A 12-step cap. Mostly so that a model stuck repeating itself can't waste inference calls without limit.
- A trajectory for every run, so I can look at what happened in a failed run instead of guessing.
donedoesn't count as success. The model reporting that it finished doesn't show that the document exists. The saved.odtstill has to pass the evaluator, and that's the next thing I wired in.
I care most about that last decision. If the model's own done became the reward, a model would learn to say done, and this loop would reward it for that.
What the first runs did
The first complete run (trajectory) used Claude Haiku, and it took six steps:
- Click at
(33, 324). That's the LibreOffice Writer icon in Ubuntu's dock, found from the screenshot alone. - Type
Hello from my Python computer-use executor!. - Press Ctrl+Shift+S to open Save As.
- Type the full target path,
/home/sarthak/cua-learning/runs/20261113-021049-9a5678/document.odt. - Press Return.
- Return
done.
It reached the same outcome as my script by a different route. The script launches Writer from the command line, presses Ctrl+N for a new document, and presses Ctrl+L before typing the path. The model clicked the dock icon, typed into the blank document Writer opened with, and typed the whole path straight into the file name field. Whether that saved the file where the evaluator expects it is the question the evaluator answers. This run happened before the evaluator was part of the loop, so it counts as "the agent declared done" and not as a pass.
I recorded a later run, 20261113-030101-7234db, with the VM on the left and the agent's terminal on the right. It's the recording at the top of this post, repeated in Figure 7 below. It took seven steps instead of six. After opening Writer from the dock, Claude clicked inside the page at (475, 340) before typing, then saved the file the same way as the first run. Each Claude call took between 4.1 and 6.7 seconds, about 35 seconds across the run.
The final done came with an explanation in its text field. Claude read the window title, document.odt, and checked the status bar's count of 6 words and 43 characters against the expected sentence. It then noted that the full path was "not visually confirmed; it relies on the earlier Save As entry." That's the same gap the loop accounts for by not treating done as success: from a screenshot, the model can see that the document was saved but not where it was saved.
The second run's trajectory stops after one action, with its status still running. The loop only sets a final status when it exits normally or catches an Exception, and a Ctrl+C raises KeyboardInterrupt, which isn't an Exception. The finally block still saved the trajectory, but its status can't distinguish an interrupted run from one that's actually in progress. Another trajectory from that night, 20261113-030034-d6b30e, has no actions at all and the same running status. That's an easy fix, and the kind of thing that matters once there are thousands of trajectories to sort.
Scoring the agent's runs on the Mac
At this point an agent run ended with Claude saying done, and nothing checked whether the document was right. Issue #23 reused the evaluator from the scripted harness to close that gap. The flow I sketched in the issue has five steps: Claude finishes its GUI actions, the harness copies document.odt out of Ubuntu, the Mac evaluates it, result.json is saved next to trajectory.json, and the exit code comes from the evaluation.
A validator bug that looked like a model failure
The first agent run that day (trajectory) clicked three times and then stopped at step 4 with status agent_error and the message Invalid Y coordinate: 419. A y of 419 is inside a screen 600 pixels tall, so the coordinate in the message was fine. The bounds check in validate_action() had a typo:
if not (0 <= x < screen_width): raise ValueError(f"Invalid X coordinate: {x}")
if not (0 <= x < screen_height): raise ValueError(f"Invalid Y coordinate: {y}")
The second line compared x with the screen's height. On the 1008×600 desktop, that rejected every click with an x of 600 or more, roughly the right two-fifths of the screen, and reported it as a bad y. It also never checked y, so a click below the bottom edge would have reached xdotool. Since 419 is in range, the click Claude proposed at step 4 must have had an x outside 0 to 599. The fix (fdae241) changes the second x to y.
From its status alone, this run reads as the model proposing an illegal action. The click was on the screen, and my validator rejected it. If errors like this were counted against the policy during training, the model would be penalized for clicking on the right side of the screen, and nobody would see why from the reward curve. This is the attribution problem from the start of this post, showing up in the layer that sits between the model and the VM.
Evaluating on the Mac
Before wiring anything in, I tested the old evaluator against a model-driven run by hand. A scratch script (1733379) copied run 20261114-172440-c3fa71 to the Mac with collect_artifacts() and called evaluate() on its document.odt. The evaluator itself needed no changes. It takes a task name, a path, and the expected text, and it has no idea whether a script or a model produced the file, which is the separation I set up in the first week.
PR #24 then put the same steps at the end of run_agent(). When the loop ends, either because Claude returned done or because it used all 12 steps, the harness:
- Copies the guest's run directory to
collected_runs/on the Mac withcollect_artifacts(), the samescpcall the scripted harness uses. - Runs
evaluate()on the Mac against the copieddocument.odt. - Writes
result.jsonnext totrajectory.jsoninagent_runs/<run_id>/. - Exits with 0 only if the run passed.
result.json has its own status, separate from the agent's: collection_failed if the copy fails, evaluation_failed if the evaluator raises or the run doesn't pass, and passed. A run passes only when two things are true:
result["end_to_end_success"] = (
result["evaluation_success"] and status == "agent_declared_done"
)
The evaluator decides whether the task was done, and the model's done can't make a run pass by itself. The same condition also fails a run that saved the right document but used up all 12 steps without returning done. For evaluation I think that's the right call, because an agent that finishes a task but can't tell that it has finished keeps spending steps and inference calls after the work is done.
Running the evaluator on the Mac also changes who can affect the score. evaluator.py and result.json only exist on the host, so nothing running inside the guest can edit the check or write its own result. The guest still produces the document, and the document is the thing being judged.
It also answers the question Claude kept raising in its own done messages. In run c3fa71 it wrote that the full path was "not confirmed visually" and relied on the earlier Save As step. The evaluator looks for document.odt in the run directory it copied, so a document saved anywhere else shows up as missing and the run fails.
The first run scored this way (trajectory, result) took six steps, the same route as the first run on November 13: a click in the dock, the sentence, Ctrl+Shift+S, the full path, Return, and done. The evaluator found the file, found exactly the one expected paragraph, and the run passed.
From a clean snapshot to a score
The last commit of the day (62222ec) joined the clean-run work from the previous chapter to the agent loop. It adds --clean-agent-run:
def run_agent_from_clean_snapshot():
reset_vm()
wait_for_ssh()
wait_for_gui()
return run_agent()
and run_agent() now ends by asking UTM to shut the guest down with utmctl stop --request. reset_vm() refuses to restore a snapshot unless the VM is stopped, and until now I had to shut it down by hand after every run. With the shutdown at the end of each run, one command covers the whole episode:
python3 agent_probe.py --clean-agent-run
The run committed with this change (trajectory, result) took eight steps. Claude clicked three times before typing, at (640, 400), (32, 325) and (771, 452), then typed the sentence, saved with the same Ctrl+Shift+S, path and Return sequence as the earlier runs, and returned done. The evaluator passed it. The third click has an x of 771, so the validator would have rejected it before the fix earlier that day.
That makes two scored agent runs, both passes, on one task. Two runs say little about how reliable the agent is, but they show that the full path from a clean snapshot to a result.json works. The next step is to run that command many times and count the passes.
How this maps to the real problem
Each piece of this repo is a small version of something a computer-use team needs at much larger scale:
- The VM and its snapshot are the environment. At scale this becomes a fleet of images that start in seconds and reset between episodes.
xdotoolandscrotare the action and observation interface. A model's action space (click at x, y; type text; press keys) has to map onto something like this, and its observation is the screenshot.- The evaluator is the verifier, and for RL it is the reward. If it is loose, like my first substring check, a model will learn to satisfy the loose version. For agent runs it now runs on the host, outside the environment the policy acts in.
- The run directory,
results.json, the agent runs'result.json, and the exit codes are what let a scheduler run many episodes and aggregate the outcomes without a person reading logs. - The two time limits keep one stuck episode from holding a VM forever or leaking into the next episode. With thousands of rollouts, some will hang, and each needs a definite end and a cause.
- The snapshot restore and readiness checks are environment reset. Each episode starts from the same disk and doesn't begin until the desktop can actually take input.
- Collected run directories and trajectories are the data. A screenshot paired with the action taken from it is the basic unit of a computer-use trajectory, whether it's used for evaluation, for debugging, or as training data.
- The action schema and validator are the action space and its safety boundary. The model proposes and the harness decides what runs. A bug in this layer shows up as a model failure, as the bounds-check typo did.
- The host controller is the start of separating the thing being tested from the thing doing the testing.
That last point used to be the biggest limitation, and for agent runs it's now handled. The scripted harness still evaluates inside the same guest it operates, where a capable enough agent with a shell could edit evaluator.py or write its own results.json. Agent runs are scored on the Mac against a copy of the run directory, so evaluator.py and result.json are out of the agent's reach whatever its action space is. What the agent can still influence is the document, which is the outcome being measured. Its effects are also still narrow: it reads screenshots and can only press seven keys, type two allowed strings, and click. That narrowness now mostly protects the VM itself, and it will matter again as the action space widens.
What's in progress
- Run
--clean-agent-runmany times back to back and measure how often the agent passes. - Give interrupted agent runs their own status instead of leaving them as
running. - Check SSH and X11 readiness once per run instead of before every screenshot, after measuring what the per-step checks cost.
- Look up the guest IP through
utmctl ip-addressand derive the GUI paths from the user's real UID. - Handle SIGTERM in
run_task.pyso a run that hits its time limit still writesresults.jsonandevents.json. - Replace fixed sleeps with waits on actual conditions, such as a window existing or a dialog closing.
- Add more applications and tasks, each with its own outcome-based evaluator, and widen the action allowlists into a real policy.
The repo, issues, pull requests, and review threads are all public at github.com/gsarthakdev/cua-learning. Each issue has the notes, screenshots, and reasoning behind the step it covers, including the parts I got wrong.