Computer Use

When an agent needs eyes and a mouse — a browser, a GUI app, anything graphical — reach for the desktop-ubuntu template. It’s a full Linux desktop you can watch and drive from the browser, with shell-driven automation available inside the image.

Create a desktop sandbox

const sandbox = await rt.sandbox.create({
  template: 'desktop-ubuntu',
  name: 'browser-env',
});

await sandbox.waitForReady();

sandbox.on('status', (state) => {
  if (state.vncReady && sandbox.vncUrl) {
    console.log(sandbox.vncUrl);
  }
});

runtools sandbox create --name browser-env --template desktop-ubuntu
runtools sandbox get browser-env

waitForReady() waits for SSH readiness. Desktop access has its own readiness state, so wait for vncReady and vncUrl before starting visual automation.

Link An Agent

agents/browser-agent.ts

import { defineAgent } from '@runtools-ai/sdk';

export default defineAgent({
  slug: 'browser-agent',
  name: 'Browser Agent',
  model: 'claude-sonnet-4',
  executionMode: 'in_sandbox',
  sandbox: 'browser-env',
  tools: ['exec_command', 'write_stdin', 'apply_patch', 'view_image', 'web_search', 'get_dev_url'],
  systemPrompt: `Use the desktop tools available through shell commands when a task requires browser or GUI inspection.`,
});

runtools deploy --agents-only
runtools agent run browser-agent "Open a browser, inspect the page, and summarize what you find"

Browser Desktop URL

The SDK exposes a normalized sandbox.vncUrl when desktop access is ready:

sandbox.on('status', (state) => {
  if (state.vncReady) {
    console.log(sandbox.vncUrl);
  }
});

The REST GET /v1/sandboxes/{id} response can also include vncReady and vncUrl. Desktop URLs are normalized to /vnc.html?autoconnect=true&resize=remote before the SDK and CLI display them.

Automation Pattern

Agents currently use shell-accessible tools for desktop automation. For example:

await sandbox.exec('xdotool getmouselocation');
await sandbox.exec('scrot /tmp/screen.png');

The desktop-ubuntu image includes XFCE, Firefox ESR, KasmVNC, xdotool, scrot, and ImageMagick. Use exec_command or sandbox.exec() for GUI-driving commands, then use a vision-capable model and view_image when your workflow depends on screenshot interpretation.

Best Practices

Prefer APIs for structured integrations

Use Tool Hub or direct APIs when a provider offers a reliable API. Desktop automation is best for web or GUI workflows that do not have a usable API.

Wait for readiness

Wait for both sandbox readiness and desktop readiness before starting browser tasks.

Keep prompts concrete

Give the agent specific page goals, selectors, or visual landmarks when possible.

Clean up long sessions

Pause or destroy desktop sandboxes when you are done.

Getting Started

Platform

Core Features

Advanced

Create a desktop sandbox

Link An Agent

Browser Desktop URL

Automation Pattern

Best Practices

​Create a desktop sandbox

​Link An Agent

​Browser Desktop URL

​Automation Pattern

​Best Practices

Create a desktop sandbox

Link An Agent

Browser Desktop URL

Automation Pattern

Best Practices