blade-computer-use

blade-computer-use is a small, agent-neutral macOS computer-use implementation over the Model Context Protocol (MCP). It gives Codex, Claude Code, Orca, and other MCP clients the same six tools:

• list_apps
• observe
• click
• type_text
• press_key
• scroll

It is an independent security research project, not an OpenAI product and not a reproduction of OpenAI's private Computer Use implementation.

Coding agent
    | stdio MCP: schemas, allowlist, one-shot revisions, PNG responses
    v
TypeScript server (bundled as one runtime file)
    | JSONL request/response protocol
    v
Persistent Swift helper
    | AXUIElement          | ScreenCaptureKit       | CGEvent
    v                      v                        v
Accessibility tree     window screenshot      mouse/keyboard input

observe returns a bounded AX tree, a window screenshot, and an opaque revision. That revision permits exactly one action. The next action consumes it, so an agent must observe again before doing anything else. element_index values are scoped to that observation and are never treated as stable cross-frame identities.

• macOS 14 or newer
• Node.js 20 or newer
• Swift 5.10 or newer and the Xcode Command Line Tools (xcode-select --install)
• Accessibility and Screen Recording permission for BladeComputerUseHelper or the agent host macOS attributes it to

The first plugin launch compiles the native helper for the current Mac. The TypeScript MCP server is already committed as a standalone bundle, so runtime installation does not run npm install.

Install this repository as a Codex marketplace, then install the plugin:

codex plugin marketplace add echoVic/blade-computer-use
codex plugin add blade-computer-use@blade-computer-use

Restart the Codex app after installation. The packaged configuration allows only TextEdit by default.

For a custom allowlist, clone the repository, change CUA_ALLOWED_APPS in .mcp.json, build once, and add the local checkout as the marketplace source:

git clone https://github.com/echoVic/blade-computer-use.git
cd blade-computer-use
npm install
npm run build
codex plugin marketplace add "$PWD"
codex plugin add blade-computer-use@blade-computer-use

Use comma-separated bundle identifiers, for example:

"CUA_ALLOWED_APPS": "com.apple.TextEdit,com.apple.Notes"

CUA_ALLOW_ALL_APPS=1 is available for local research, but the lock screen and SecurityAgent remain hard-denied.

Clone the repository, then load it as a local Claude Code plugin:

git clone https://github.com/echoVic/blade-computer-use.git
claude --plugin-dir /absolute/path/to/blade-computer-use

Claude Code reads .claude-plugin/plugin.json and the shared root .mcp.json. ${CLAUDE_PLUGIN_ROOT} is resolved by the launcher, so the checkout can live anywhere.

Copy the Orca MCP example into Orca's configuration and replace the absolute command path. The example uses an underscore-safe server name, so Orca discovers the tools as mcp__blade_computer_use__<tool>.

The first observe requests Accessibility and Screen Recording through the macOS system APIs. The user must approve both permissions; the plugin cannot toggle security settings on the user's behalf.

If macOS opens System Settings > Privacy & Security, grant:

1. Accessibility to BladeComputerUseHelper, or to the terminal/Codex/Claude host shown by macOS.
2. Screen Recording to the same executable shown by macOS.

Restart the plugin host after changing permissions, then retry observe. list_apps reports both accessibility_trusted and screen_recording_trusted; a denied request returns permission_denied with the missing permission names.

1. Open TextEdit and create a new plain-text document.
2. Ask the agent to call list_apps and find com.apple.TextEdit.
3. Ask it to observe TextEdit.
4. Ask it to click the editor, type a short line, and verify the result.

The expected loop is observe -> one action -> observe. A stale or already-consumed revision must not be retried.

With an empty TextEdit document focused and both permissions granted, run the live integration check:

npm run test:manual:textedit

This opt-in test verifies AX observation, two PNG responses, Unicode input, one-shot revision rejection, and the final AX text. It is not part of CI because it requires an interactive macOS session and user-granted permissions.

• Apps require an explicit bundle-id allowlist; TextEdit is the only packaged default.
• com.apple.loginwindow, com.apple.SecurityAgent, and com.apple.SecurityAgentHelper are always denied.
• Secure AX fields reject type_text.
• Coordinate actions verify that the observed window has not moved or resized.
• Screenshots are written to a mode-0700 temporary directory, read into the MCP image response, and deleted immediately afterward.
• No telemetry is implemented. AX text and screenshots are returned to the connected agent, whose own model-request and retention behavior is outside this project.

See SECURITY.md for the trust boundary and reporting process.

This is deliberately smaller than a production computer-use stack:

• macOS-only; there is no Linux or Windows implementation.
• No AX diff protocol or cross-observation element identity matching.
• No lock-screen guardian, URL policy, or physical user-intervention detector.
• No OCR, browser-specific DOM bridge, or action-specific confirmation UI.
• No telemetry, account-policy integration, code signing, notarization, or prebuilt native binaries.
• The AX tree is capped at 500 nodes, depth 12, and 160 characters per text field.

npm install
npm test
npm run typecheck
npm run build
npm run validate:manifests
npm run smoke:mcp

The native checks use a framework-free self-test executable because a Command Line Tools-only Swift installation does not provide XCTest or the Swift Testing module:

npm run test:native

The self-test currently covers 22 protocol, AX serialization, input, coordinate, and permission-gate checks.

MIT