This article was written by AgentOne Research.

We just open-sourced computer-use-rs, a cross-platform Rust crate that lets your code look at any desktop app and use it the way a person would. It can read what's on screen, take a screenshot, and click, type, scroll, and drag.

It's the same library that powers Computer Use in the AgentOne desktop app. It's free, it works on macOS, Windows, and Linux, and it's dual-licensed under MIT or Apache-2.0.

cargo add computer-use-rs

What Is "Computer Use"?

Most AI agents work through APIs. If you want an agent to send an email, someone has to build an email tool for it. That works great for popular apps, but most of the software on your computer doesn't have an API at all.

Computer use is the other approach: let the agent operate the computer directly. It looks at an app, figures out what's there, and clicks and types like you would. If you can do it with a mouse and keyboard, an agent with computer use can (at least in theory) do it too.

To make that work, the agent needs two things:

1. A way to see. What app is open, what buttons and text fields are in it, and what it looks like.
2. A way to act. Clicking, typing, pressing keyboard shortcuts, scrolling, and dragging.

computer-use-rs is both of those, in one Rust crate.

What computer-use-rs Does

Everything goes through one type, Computer. Here's a quick example that types into TextEdit on a Mac:

use computer_use::{CaptureOptions, ClickOptions, ClickTarget, Computer, ElementQuery};
 
fn main() -> computer_use::Result<()> {
    let mut computer = Computer::new()?;
 
    // Observe: an indexed element tree, a PNG screenshot, and a plain-text outline.
    let state = computer.get_app_state("TextEdit", &CaptureOptions::default())?;
    println!("{state}");
 
    // Act by element index…
    let text_area = state.find(&ElementQuery::role("text area")).next().map(|e| e.index).unwrap();
    computer.click("TextEdit", ClickTarget::Element(text_area), ClickOptions::default())?;
    computer.type_text("TextEdit", "Hello from Rust!")?;
 
    // …or by screenshot pixel.
    computer.click("TextEdit", ClickTarget::Pixel { x: 120.0, y: 80.0 }, ClickOptions::default())?;
    computer.press_key("TextEdit", "Return")?;
    Ok(())
}

Seeing an app

One call to get_app_state gives you three views of an app's window:

• An accessibility tree, as a numbered list of elements (buttons, text fields, menus, links), with their roles, labels, values, and the actions you can take on them
• A PNG screenshot of the window
• A plain-text outline of the whole thing, which is exactly what you'd hand to a language model

If the app isn't running yet, it gets launched in the background. You can also use query to find elements by text or role without taking a screenshot, and list_apps / list_windows to see what's available.

Using an app

Once you know what's on screen, you can:

• Click an element by its index, a pixel in the screenshot, or a point on the screen (left, right, or middle, single or multi-click)
• Type text, and press key chords like ctrl+shift+Tab or super+c
• Scroll and drag
• Set values directly on settable elements, like sliders and text fields
• Select text inside an element, or put the caret before or after it
• Trigger named actions like "Show Menu", "Raise", or "Increment"

Wherever it can, computer-use-rs uses accessibility actions instead of fake mouse clicks. Pressing a button through the accessibility API is faster and a lot more reliable than guessing where to click.

It works in the background

This is the part I'm most excited about. By default, input goes straight to the target app without moving your mouse or stealing focus, wherever the OS allows it.

That matters a lot for agents. Nobody wants their cursor flying around the screen while an agent fills out a form. With background delivery, the agent can work in one app while you keep using another.

There are a few delivery modes for apps that need something different:

On macOS, Command-key shortcuts even work in the background, because they're resolved to the matching menu item in the app.

Platform Support

computer-use-rs uses each operating system's own accessibility and capture APIs:

A few things to know before you start:

• macOS: whatever runs your binary (usually your terminal) needs Accessibility and Screen Recording permissions. Computer::permissions() tells you what you have, and Computer::request_permissions() shows the system prompts.
• Linux: the accessibility bus needs to be running, which it is by default on GNOME and KDE. computer-use-rs turns on the desktop's accessibility flag for you, but Chromium, Electron, and Firefox apps only check it at startup, so apps that were already open need a restart (they're flagged with needs_restart). On Wayland, you get trees and actions for everything, but screenshots and input only work for X11/XWayland apps.
• Windows: no setup needed.

Not Just for AI Agents

We built computer-use-rs for AgentOne, but there's nothing AI-specific about it. It's a plain Rust library: no server, no runtime, no protocol, just a typed API. That makes it useful for a lot of other things:

• UI automation and scripting, for those boring, repetitive tasks in apps that don't have an API
• End-to-end tests for desktop apps
• Accessibility tools that need to read and interact with other apps

A few other details Rust developers will appreciate:

• Computer is Send, so you can put it behind a mutex and use it from spawn_blocking or any worker thread.
• AppState and Element are plain data, so they're easy to serialize, log, or pass around.
• The platform layer is a public Backend trait. You can drive a fake desktop in your tests, or add support for a new platform.
• The smaller building blocks are public too, like the xdotool-style key parser (keys::KeyChord) and the helper that maps screenshot pixels to screen points.

Why We Open-Sourced It

When we started building Computer Use for AgentOne, we went looking for a Rust library that could do this on all three platforms. We didn't find one we were happy with. Most options were tied to one OS, only handled mouse and keyboard input without the accessibility tree, or came wrapped in a server or protocol we didn't want.

So we built our own. And once it was working, keeping it to ourselves didn't make much sense.

Software that controls your computer should be auditable. A library that can read every window and click any button is about as sensitive as code gets. If we're asking you to trust an agent with that kind of access, you should be able to see exactly how it works. That's the same reason we open-sourced the AgentOne desktop app.

More users means a better library. Desktop automation is full of edge cases. Every app, toolkit, and window manager behaves a little differently, and we'll never test them all ourselves. Every bug report and pull request makes computer-use-rs better, and that makes Computer Use in AgentOne better too.

Computer use should be a building block, not a moat. Right now, most good computer-use tools live inside big AI companies' products. We'd rather everyone have access to solid, open primitives, so people can build agents, tests, and tools we haven't even thought of.

That's also why we picked MIT or Apache-2.0 instead of the AGPL license that AgentOne itself uses. We want computer-use-rs to be easy to drop into any project, open source or not.

How to Try It

Add it to your project:

cargo add computer-use-rs

The crate is imported as computer_use. If you want to poke around first, clone the repo and run the examples:

cargo run --example list_apps
cargo run --example inspect -- TextEdit /tmp/textedit.png
cargo run --example textedit   # macOS: create a document, type, select, delete

The inspect example is a good place to start. Point it at any app and it prints the full element tree and saves a screenshot, so you can see exactly what an agent would see.

Get Involved

computer-use-rs is brand new, and we'd love your help.

• ⭐ Star the repo: github.com/The-Best-Codes/computer-use-rs
• 📦 Get the crate: crates.io/crates/computer-use-rs
• 🐛 Open issues when an app doesn't behave the way you expect. These are the most helpful reports we can get.
• 🔧 Submit a PR, especially for Linux and Windows edge cases
• 💬 Come chat in the AgentOne Discord

And if you'd rather have an agent that can already use your computer, without writing any Rust, AgentOne is free to download.

Happy building!