AIYING V5.0 | AI IN ACTION
AI Improved the Average. Who Actually Benefited?
By Ravindra Dastikop
A team introduces an AI assistant. Productivity improves.
The announcement practically writes itself: “AI is helping our employees.”
But a useful question remains:
Which employees—and what happened to the quality of their work?
A documented customer-support deployment shows why that question matters.
The real-world case
In Generative AI at Work, published in The Quarterly Journal of Economics in 2025, Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the staggered introduction of an AI assistant across 5,172 customer-support agents.
The assistant suggested responses during customer chats. Human agents remained responsible and could edit or ignore its suggestions.
The researchers found a 15% average increase in issues resolved per hour. [1]
What the average concealed
The benefits varied substantially.
Less experienced and lower-skilled workers improved in speed and quality. The most experienced and highest-skilled workers saw small speed gains and small quality declines.
This was evidence from one firm’s deployment. It did not establish the overall effects of AI on employment or wages. [1]
See differently: examine who needs what
My interpretation: the practical unit of evaluation should be the worker doing a particular task, alongside the team average.
A new employee may need help recognising the problem and structuring a response. An experienced employee may need room to depart from a suggestion when the situation calls for it.
Giving everyone access is one implementation decision. Deciding how assistance should fit different people’s work is another.
Generate possibilities for your own team
The following are proposed applications, not outcomes demonstrated by this study:
| Situation | Possibility to test |
|---|---|
| New employees struggle to begin | Provide suggestions linked to checked examples and explanations |
| Experienced employees handle unusual cases | Make suggestions easy to dismiss and exceptions easy to record |
| Managers see only a team average | Review results by experience level and task type |
| Faster work may hide extra correction | Track rework alongside completion time |
Choose the possibility that addresses an observed difficulty.
A small implementation exercise
Select one recurring task, such as responding to a routine enquiry.
Record a baseline before introducing assistance. Then run a limited pilot with a comparable set of tasks.
Review:
- Time taken to complete the task.
- Accuracy and completeness.
- Corrections or rework required.
- Differences between newer and experienced participants.
Use a consistent quality checklist. Where practical, have the reviewer assess outputs without knowing whether AI helped produce them.
A small pilot can inform your next decision. It cannot establish a universal productivity gain.
What would count as an accomplishment?
Set the acceptance criteria before looking at the results.
For example:
Continue the pilot if participants complete the task faster while meeting the agreed quality standard and without increasing rework.
Then examine whether the result holds for different participants.
If newer colleagues benefit while experienced colleagues need a different workflow, that finding can guide a more useful implementation.
The transferable lesson
An average improvement is the beginning of an evaluation.
Ask where the benefit appears, what it costs and whose work needs a different arrangement.
That turns an AI success claim into an operational decision.
Your turn
If your team reported an AI productivity gain tomorrow, which result would you ask to see next: quality, rework or differences by experience?
Explain what that result would help you decide.
Source
[1] Brynjolfsson, E., Li, D., and Raymond, L. (2025). Generative AI at Work. The Quarterly Journal of Economics, 140(2), 889–942.
The findings above refer to the published journal version. The implementation exercise is a proposed application, not a replication or a completed pilot.

