GitHub Copilot can now control desktop applications on macOS and Windows

Currently available as a public preview in Copilot CLI and the Copilot app, this feature reads, clicks and types within software that lacks an API, following the user’s explicit consent.

GitHub opened a computer control function for Copilot on October 1, in public preversion. Available in Copilot CLI and GitHub Copilot on macOS and Windows, it allows the wizard to read accessible content and display an application, click on its commands and type or correct text.

For software without APIs

GitHub highlights the automation of old or only graphic software, which does not offer a command line interface or API. Copilot only takes control of an application after the user agrees; On Mac, it guides the activation of access and screen recording permissions, and the list of approved applications can be reviewed or reset at any time.

Organizations can disable the function by their administration settings. GitHub recommends describing the expected result, the applications involved and the constraints to be respected rather than a sequence of clicks.