Computer

​

Key Features

• Screenshots - Capture current screen state
• Mouse Control - Move, click, drag operations
• Keyboard Input - Type text, press keys
• Screen Navigation - Scroll, zoom, multi-window
• Real-time Feedback - Visual loop with agent decisions
• State Tracking - Remember screen positions

​

How to Enable

​

For Agents Only

1. Agents > Select Agent > Settings > Capabilities
2. Search for Computer
3. Click Attach
4. Save

​

Usage Examples

​

Take Screenshot

"Take a screenshot of the current desktop"

​

Click on Element

"Click the blue button in the top-right corner"

1. Analyzes screenshot
2. Identifies button coordinates
3. Executes click
4. Captures new screenshot

​

Fill Form

"Fill the login form with username 'alice' and password 'secret'"

1. Screenshots form
2. Identifies input fields
3. Clicks on username field
4. Types username
5. Clicks on password field
6. Types password
7. Clicks submit button

​

Navigate Multi-step Process

"Open the application, go to settings, and change the theme to dark"

​

Screen Coordinates

Screenshot resolution: 1920x1080
Button position: x=1850, y=20

• Click at coordinates
• Drag between points
• Identify text positions
• Calculate relative positions

​

Interaction Types

​

Mouse Actions

• click(x, y) - Single click
• double_click(x, y) - Double click
• right_click(x, y) - Right/context click
• drag(x1, y1, x2, y2) - Drag from point to point
• move(x, y) - Move cursor without clicking
• scroll(direction, amount) - Scroll up/down/left/right

​

Keyboard Actions

• type(text) - Type text string
• key(name) - Press single key (Enter, Tab, Escape, etc.)
• hotkey(mod, key) - Keyboard shortcut (Ctrl+C, Cmd+V, etc.)

​

Navigation

• screenshot() - Capture current screen
• wait(seconds) - Wait for page to load
• maximize() - Maximize window
• minimize() - Minimize window

​

Size Tiers

​

Supported Operating Systems

• Ubuntu 24.04, Ubuntu 22.04
• Debian 12, Debian 11

​

Configuration

{
  "computer": {
    "shell_enabled": true,
    "browser_enabled": false,
    "browser_max_tabs": 5,
    "browser_max_download_mb": 50,
    "browser_allowed_domains": []
  }
}

​

Browser Subsystem

​

Browser Tools

​

Stateful vs Stateless Browser

​

Domain Allowlist

{
  "browser_allowed_domains": ["example.com", "app.internal.com"]
}

​

Resource Limits

​

Cost

​

Common Use Cases

​

Web Application Testing

Screenshot app, verify buttons, test workflows

​

Automation

Automate repetitive UI tasks programmatically

​

Data Entry

Fill forms and navigate multi-step processes

​

Visual Inspection

Verify visual appearance matches requirements

​

Agent Loop Pattern

1. Screenshot - See current state
2. Analyze - LLM processes image
3. Decide - LLM decides next action
4. Execute - Perform mouse/keyboard action
5. Repeat - Loop until task complete

​

Best Practices

​

Start with Screenshot

​

Be Explicit

✓ "Click the 'Save' button (blue, bottom-right)"
✗ "Save the file"

​

Handle Errors

"Screenshot again to verify action completed"

​

Use Coordinates When Possible

"Click at coordinates (1850, 50)"

​

Wait for State Changes

"Wait 2 seconds for dialog to load"
"Take screenshot to verify"

​

Limitations

• Desktop/Web only - Works with rendered interfaces
• Not for APIs - Use HTTP Client for APIs
• Visual interpretation - Relies on screenshot analysis
• Speed - Slower than direct API calls
• Flakiness - UI changes can break workflows

​

When NOT to Use

​

Troubleshooting

​

Screenshot is blank

• Wait for page to load
• Check window is focused
• Verify viewport size is correct

​

Click doesn’t work

• Coordinates may be off
• Element may not be clickable
• Try right-clicking instead
• Screenshot again to verify state

​

Text not entered

• Field may not be focused
• Type more slowly
• Use keyboard navigation (Tab)
• Copy-paste if typing fails

​

Agent stuck in loop

• Break task into smaller steps
• Increase wait times
• Provide more explicit instructions
• Use timeout to stop execution

​

Privacy & Security

• Use dedicated user accounts
• Limit to non-sensitive applications
• Monitor screen capture content
• Disable in production where possible

​

API Access

# Execute computer action
curl -X POST https://api.noorle.com/v1/agents/{agent_id}/computer \
  -H "X-API-Key: ak-{your_key}" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "screenshot"
  }'

# Click action
curl -X POST https://api.noorle.com/v1/agents/{agent_id}/computer \
  -H "X-API-Key: ak-{your_key}" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "click",
    "x": 100,
    "y": 100
  }'

​

Next Steps

• Browser Capability — Stateless browser for quick page rendering
• Web Search - Find Information
• HTTP Client - API Access
• Code Runner - Process Data
• Creating Agents