Task 1
Table of Contents
Interview colleagues
Interacting with people via Slack was surprisingly easy for the AI
Task 2
Suggest staff cuts
The AI completed the task but made a fatal error.
Task 3
Complete 17 forms
It mastered the tedious part and struggled with a trivial step.
We gave an AI tool full access to a laptop with pre-configured apps and tried to answer a simple question: Can artificial intelligence do an office job?
Some executives seem to think so. According to Layoffs.fyi, an industry tracking website, more than 200 tech companies have cut around 120,000 jobs this year; Meta, Oracle and others have all made significant staff cuts recently and cited AI as a driving force; and after laying off about 1,100 employees, Cloudflare’s chief executive recently said he expects AI to replace middle management, finance and marketing employees.
In our experiment, we used AI “agents” as office workers and found that they were able to perform some, but not all, of the tasks we assigned. The agents, capable of acting autonomously and making decisions based on detailed instructions, excelled at solving problems by writing computer programs. But they struggled to understand the nuances of human language and navigate user interfaces like the Chrome web browser.
The first task we gave our AI was to collect feedback from colleagues on the messaging app Slack and then enter their responses into a spreadsheet. We created Slack bots that act as colleagues and answer (or don’t answer) the agent’s questions with pre-written answers.
During this task, the agent managed several curveballs without missing a beat. When his colleague sent him the list of meeting participants, one of the names was misspelled (“Sara Johnson” instead of “Sarah Johnson”), but the agent still sent a message to the correct person. One of the participants who was sent the message never responded, and instead of having to wait, the agent simply added that employee to the table without any points.
This task and the others we assigned to AI were adapted from papers and benchmarking tools recently published by researchers at Carnegie Mellon University and OpenAI. The researchers designed the benchmarks to test the performance of different models – such as OpenAI’s GPT, Google’s Gemini and Anthropic’s Claude – in real-world environments and compare them with each other.
Both research projects included extensive open-source agent testing materials, including fabricated spreadsheets, memos, personnel lists, and other company documents, as well as carefully written prompts designed to provide agents with the necessary context and explain exactly what role they played and what tools were available to them.
We didn’t use the benchmark materials to compare models, but rather to get a general sense of what agents are capable of when given expert guidance and supporting documentation. We ran the agents using Anthropic’s Claude Cowork app, which allowed them to take control of the computer and applications.
The next task we gave our AI was to identify staffing cuts to meet the budget needs of a government agency. The agent had to review several documents, decide which roles in the agency could be eliminated, and prepare a three-document report of his conclusions.
In this case, the AI managed the series of tasks and realized that the budget targets could be met without layoffs because several employees planned to retire or resign before the next fiscal year.
But it also made a serious mistake: When it became clear that at least three employees wanted to take leave, it added their positions to the list of positions that could be eliminated for the following year, without taking into account when those employees wanted to return.
While a human would instinctively ask how long these vacation days would last, the AI concluded from this section that all of these employees could be lumped together and written off.
“AI may be less able to replace tacit knowledge, the idiosyncratic tips and tricks that accumulate with experience but are never digitized,” researchers at Stanford University and the National Bureau of Economic Research said in a recent study.
The agent also had difficulty navigating the Google Drive interface and a spreadsheet. And in the step where the agent was supposed to modify an existing org chart in Preview or Google Slides – a task he probably could have done – he almost immediately decided that using an app was too inconvenient and wrote code instead.
“Agents work in a very nonhuman way, writing code instead of using the typical interfaces that humans use,” said Graham Neubig, a professor at Carnegie Mellon University and co-author of the AgentCompany research project.
Not only has our agent shown a preference for programming his way through a problem; It also ignored our instructions to try using apps before switching to code. AI seems to understand that it is very good at encoding and processing structured data and less good at navigating applications designed for the human eye and intuition.
For the next task, the prompt instructed the AI to fill out 17 I-9 employment verification forms, using employee data from a spreadsheet and uploading them to a Google Drive folder.
Despite his instructions, the agent once again avoided using an app for work and instead started writing code after about 30 seconds. However, the programmatic solution worked well and the agent even added a section to the script to ensure that the phone and social security numbers were formatted correctly.
But after going through what would have been a laborious and time-consuming process for a human, the agent then encountered a step that would have been trivial for any office worker: uploading the PDF documents to Google Drive.
The results of our experiment were similar to what researchers and companies have found when testing and deploying artificial intelligence tools. Scale AI, an AI training company, recently tested agents on real freelance projects, and the top-scoring model only delivered client-ready work about 16 percent of the time.
While AI can regularly excel at complex tasks, it can be unreliable when tasked with responsibility for an entire task. It can certainly add value to certain areas of the workforce, but for now, AI still needs a human boss.
https://www.nytimes.com/interactive/2026/07/23/technology/ai-agents-office-jobs.html
