Generative AI at Work: Gains, Gaps, and Governance
In many organizations, the promise of introducing advanced technologies centers on automation, cost reduction, and speed. However, empirical research suggests that a key factor in the success of new technologies is not only the capability of the machine but also how humans, systems, and technologies interconnect.
A study by Brynjolfsson, Li, and Raymond titled “Generative AI at Work,” published in The Quarterly Journal of Economics, provides a rich case study that illustrates this point. The authors introduce the question of implementing AI technologies as follows:
"The emergence of generative artificial intelligence (AI) has attracted significant attention for its potential economic impact. Although various generative AI tools have performed well in laboratory settings, questions remain about their effectiveness in the real world, where they may encounter unfamiliar problems, face organizational resistance, or provide misleading information (Peng et al. 2023a; Roose 2023)."
The Study in Brief
They examined the implementation of a generative AI conversational assistant among 5,172 customer support agents “working for a Fortune 500 firm that sells business-process software.” Regarding the focus of their study, they write:
"We study the effect of generative AI on productivity and worker experience in the customer-service sector, an industry with one of the highest surveyed rates of AI adoption (…). The tool we study is built on Generative Pre-trained Transformer 3 (GPT-3), a member of the GPT family of large language models developed by OpenAI (OpenAI 2023). The AI system monitors customer chats and provides agents with real-time suggestions for how to respond. The AI system is designed to augment agents, who remain responsible for the conversation and are free to ignore or edit the AI’s suggestions."
They found that access to the AI tool increased productivity, defined as issues resolved per hour, by approximately 15% on average. Importantly, the gains are not uniform: less experienced and lower-skilled agents show the largest relative improvements, whereas the most experienced / high-skilled agents show little change in productivity and small declines in quality. They explain:
"The gains accrue disproportionately to less experienced and lower-skill customer-support workers indicating that generative AI systems may be capable of capturing and disseminating the behaviors of the most productive agents."
Productivity Gains Are Heterogeneous
The broad headline — a 15% increase — is encouraging. However, a closer look at the study reveals that the gains are concentrated among workers with lower initial performance levels. For these workers, AI assistance acts as a knowledge and skills amplifier, helping them replicate the strategies of high-performing peers and speed up the learning process. The authors demonstrate that novices who received AI assistance from the beginning achieved the same output in two months as agents who had worked without AI assistance for more than six months.