Computer use in Codex
Codex has evolved from a coding agent into a versatile teammate capable of controlling any GUI application on your Mac via computer use. It uses a combination of multimodal models and accessibility frameworks to interact with apps, allowing it to perform tasks like creating virtual machines, playing music, and setting reminders, all while you continue working. The feature prioritizes safety by requiring per-app permission and enables superhuman speed with models like Spark.
Key points
Detailed summary
Introduction
Roma introduces Codex's evolution from a coding agent to a teammate capable of computer use, which allows it to interact with any local app via GUI. Ari joins to explain the feature.
Why Computer Use?
Ari explains that Codex could already run commands and write code, but computer use extends its capabilities to any graphical interface, enabling it to click, type, and navigate apps like a human.
Onboarding Experience
The onboarding is simple: Codex asks for permission, animates the settings panel, and guides the user through two drags to authorize system changes.
Demo: Creating a Virtual Machine
Ari demonstrates creating a Mac VM in UTM. Codex launches UTM, clicks through the setup, and downloads macOS—all while the user can continue working.
Multitasking with Multiple Apps
Ari shows Codex playing music in Spotify and adding a reminder in the Reminders app simultaneously. Codex uses multiple cursors across apps without interrupting the user.
Cursor Design
The cursor motion is designed to feel natural and whimsical, with curves that make it look like it's swimming across the screen, helping users understand what the agent is doing.
Multimodal and Accessibility Integration
Codex combines multimodal models (screenshots) with accessibility framework data to understand interfaces deeply, even elements scrolled off-screen. This allows using fast non-multimodal models like Spark for superhuman speed.
Demo: Superhuman Speed with Spark
Using the Spark model, Codex sends a message in Messages faster than a human could, demonstrating its ability to perform tasks at superhuman speed.
Future of Computer Use
Ari discusses how computer use capabilities are now integrated into main GPT models, enabling faster development. The goal is to make computer use 2-10x faster than humans, becoming indispensable for all computing tasks.
Safety Approach
Safety is paramount: Codex requires permission for each app it accesses and cannot see or interact with unapproved apps. This builds trust by limiting access to sensitive applications.
Personal Use Cases
Ari uses Codex to update financial spreadsheets, saving time. Roma notes that computer use fills the gap of accessing local apps, making Codex an end-to-end assistant.
Conclusion
Computer use transforms how your computer works, doing real work in the background without breaking your flow. It's available for Mac today, with Windows support coming soon.
Transcript
Summarize your own video
Paste any YouTube link and get a TL;DR, key points and the full transcript like this one — in seconds.