Computer-Using Agent

1 - What is "Computer-Using Agent"

OpenAI has developed a new artificial intelligence model called the Computer-Using Agent. This model is designed to interact with computers and the internet in a way that mimics human behavior, allowing it to perform tasks such as browsing websites, filling out forms, and making online purchases.

Key Features of CUA:

• Vision Capabilities: CUA can process and understand visual information, such as images and screenshots. This enables it to navigate graphical user interfaces and interpret visual data effectively.

• Advanced Reasoning: Beyond basic interactions, CUA possesses advanced reasoning skills, allowing it to make informed decisions, solve complex problems, and adapt to various tasks.

Integration with Operator:

CUA powers OpenAI's Operator, an AI agent capable of autonomously performing tasks on the web. Operator can handle activities like booking travel, purchasing groceries, and managing schedules by interacting with websites as a human would. It captures screenshots to monitor the screen, analyzes those images using GPT-4o's vision capabilities, and performs actions such as typing, clicking, and scrolling.

Safety and User Control:

To ensure safe and reliable operation, CUA includes safeguards such as:

• Confirmation Prompts: Before executing critical actions, like completing a purchase or sending sensitive information, CUA requests user confirmation to prevent unintended consequences.

• Monitoring for Malicious Inputs: CUA is equipped to detect and mitigate attempts to manipulate its behavior through malicious inputs, ensuring secure interactions.

In summary, the Computer-Using Agent represents a significant advancement in AI technology, enabling machines to interact with computers and the internet in a human-like manner. This capability opens up new possibilities for automating complex tasks and enhancing productivity.

2 - OpenAI launches Operator, an AI agent that can operate your computer

Benj Edwards – Jan 23, 2025

New research "Computer-Use Agent" AI model can jump in and help users with on-screen tasks.

On Thursday, OpenAI released a research preview of "Operator," a web automation tool that uses a new AI model called Computer-Using Agent (CUA) to control computers through a visual interface. The system performs tasks by viewing and interacting with on-screen elements like buttons and text fields similar to how a human would.

Operator is available today for subscribers of the $200-per-month ChatGPT Pro plan at operator.chatgpt.com. The company plans to expand to Plus, Team, and Enterprise users later. OpenAI intends to integrate these capabilities directly into ChatGPT and later release CUA through its API for developers.

Operator watches on-screen content while you use your computer and executes tasks through simulated keyboard and mouse inputs. The Computer-Using Agent processes screenshots to understand the computer's state and then makes decisions about clicking, typing, and scrolling based on its observations.

OpenAI's release follows other tech companies as they push into what are often called "agentic" AI systems, which can take actions on a user's behalf. Google announced Project Mariner in December 2024, which performs automated tasks through the Chrome browser, and two months earlier, in October 2024, Anthropic launched a web automation tool called "Computer Use" focused on developers that can control a user's mouse cursor and take actions on a computer.

"The Operator interface looks very similar to Anthropic's Claude Computer Use demo from October," wrote AI researcher Simon Willison on his blog, "even down to the interface with a chat panel on the left and a visible interface being interacted with on the right."

Watch and take action

To use your PC like you would, the Computer-Using Agent works in multiple steps. First, it captures screenshots to monitor your screen, then analyzes those images (using GPT-4o's vision capabilities with additional reinforcement learning)

Episode 3916 - Jan 25 - Tiếng Anh - Tác nhân Sử dụng Máy tính - Vina Technology at AI time

Vina Technology at AI time - Công nghệ Việt Nam thời AI

Episode 3916 - Jan 25 - Tiếng Anh - Tác nhân Sử dụng Máy tính - Vina Technology at AI time