Skip to content
OriginalAnalysisAI

Gemini can now use your Mac for you

Google's computer use lets Gemini click, type and fill in forms across Mac apps and the web, in a background window while your cursor stays yours. The catch is who gets it — and what Apple thinks of it.

A black-and-white Op Art print of rippling wavy stripes filling the whole frame, with one flat red mouse-pointer arrow near the centre and one small grey mouse-pointer arrow near the lower-left edge
Two cursors in one picture: the red one is at work inside the pattern, the grey one is at rest.
On this page

In brief

  • Computer use, live November 19 in the Gemini app for Mac, lets Gemini click, type and fill in forms across local files, desktop apps and the web, in a background window while your screen stays free.
  • You start a task by adding @Computer use to a prompt. A new Desktop group under Customize > Connectors lists Blender and Autodesk Fusion alongside the existing Git and Obsidian integrations.
  • Rollout is US English only, for personal accounts 18 and over on Apple Silicon Macs with macOS Sequoia 15.0+, on Google AI Plus, Pro or Ultra. Support docs reportedly name only Pro and Ultra.
  • Gemini pauses for logins and payments and needs confirmation for sensitive actions, but Google's own support docs warn agents can meet misleading instructions inside apps and websites.
  • Apple tightened the other side in macOS 27.3 beta 2 on November 16: users pick exactly which apps an automation tool may control, with a menu bar indicator while one is in control.

Google gave Gemini the run of your Mac yesterday, and the demo it chose is a child's drawing of a butterfly. Hand Gemini the sketch, add "@Computer use" to your prompt, and watch it open Blender, model the butterfly, and rig it for animation, all in a window you never touch.

That is the pitch for computer use, the feature Google switched on November 19 in the Gemini app for Mac. Gemini stops being a chatbot that answers questions and becomes a pair of hands. It clicks, types, opens menus, and fills in forms across your local files, desktop apps, and the web, carrying multi-step jobs from start to finish while you keep working.

The window you never see

The trick that makes the feature usable is where it happens. Gemini works inside an isolated background window, with a live preview of what it is doing right inside your chat. Your own screen and your cursor stay free. You can keep typing, designing, or taking a call while the model color-grades a batch of photos or pulls numbers out of a spreadsheet somewhere behind you.

Google's launch post names four jobs it thinks the feature is for: batch color-grading photos or setting up complex 3D scenes; pulling weights, dimensions, and addresses out of local PDFs to fill in multi-box shipping quotes; gathering metrics and screenshots from live dashboards across regions and building a presentation in Google Slides; and analyzing clothing photos from a local folder, researching prices, and staging resale listings.

This is a step beyond what the Mac app could already do. The existing Task feature, previously called Gemini Spark, could reach into local files and pull in connected apps like Gmail or Google Calendar. Computer use is different in kind. It drives the interface itself, in third-party apps Google does not control, the way a person sitting at the keyboard would.

To start a task you add "@Computer use" to your prompt. Under Customize, then Connectors, a new "Desktop" group appears, with apps like Blender and Autodesk Fusion listed alongside the Git and Obsidian integrations that were already there. The list reads like a statement of intent: the creative tools first, the places where repetitive clicking eats whole afternoons.

The gate

Not everyone gets in. Computer use is rolling out in English in the United States, to personal Google accounts belonging to users 18 and over, on Apple Silicon Macs running macOS Sequoia 15.0 or later, with a Google AI Plus, Pro, or Ultra subscription. The plans start at $4.99 a month, according to MacRumors. The rollout is gradual, so even an eligible account may not see the feature yet.

One wrinkle is worth knowing before you pay for a plan to get it. Google's announcement lists Plus, Pro, and Ultra, but AppleInsider reports that Google's own support documentation currently names only Pro and Ultra for controlling desktop apps. Check that the feature actually appears on your account before upgrading.

Google has built in the obvious brakes. You choose which apps Gemini can reach. You can watch every step live and stop it with one click. The model pauses for logins and payments, and sensitive actions need your confirmation first. Those are real controls, and they are also the whole argument. This is a feature about trust, and the trust it asks for is not all or nothing.

There is a quieter warning in Google's support documentation, flagged by The Mac Observer: AI agents can run into misleading instructions inside the apps and websites they operate, which makes careful permission management important. An agent that reads the screen the way you do also reads whatever a webpage tells it to do. The prompt injection problem, the one that has haunted every agent that touches the browser, now has a seat at your desktop.

The landlord has opinions

Google is not alone in this race, and the timing is pointed. Anthropic's Claude already controls Mac applications, and Apple is developing its own agent that can work across apps and browse the web, according to The Mac Observer. Google's move puts a capable desktop agent in front of Mac users before Apple's own version arrives.

Apple, meanwhile, has started tightening the screws on exactly this kind of access. In macOS 27.3 beta 2, released November 16, Apple split the old Device Control and Data Access setting in two. One part covers accessibility tools like screen readers. The other, System and App Protection Override, covers apps that control other apps, and it lets you pick which apps an automation tool may touch: Mail, say, but not Messages. You can also set it to ask first every time, and a menu bar indicator appears whenever an app is controlling another one. Full Disk Access gets a sterner warning too, spelling out that granting it lets an app read, change, delete, and export your data.

Apple warned on October 2 that some developers were asking for Full Disk Access in ways that could expose everything on a machine, including mail, messages, and browsing history, without users fully understanding what they had granted. The new controls do not stop you from handing an agent the keys. They make sure you know you did.

None of this makes computer use a bad idea. The jobs Google describes are real, and the background window is a genuinely clever answer to the oldest complaint about agents: they take over your screen to save you time, then waste it while you watch. But the feature's real test is not whether Gemini can model a butterfly. It is whether the controls hold when the task is a tax form, a company dashboard, or a webpage with instructions of its own. Google has handed you an extra set of hands and a stop button. The rest is on you, which is exactly how it should be, and exactly why it is worth pausing over.

Related reporting

More from grep.tips

All articles