AI Optimism and Uncertainty: What Can Earnings Calls Tell Us Post-ChatGPT?

On Nov. 30, 2022, artificial intelligence company OpenAI released an early version of chatbot ChatGPT to the public. Since then, artificial intelligence (AI) has become an increasingly hot topic. AI brings with it potential for productivity growth, as well as uncertainty and risk for some firms. Finding ways to gauge how firms are reacting to AI can help economists understand its impact on various aspects of the economy, including productivity, firm-level investment and the labor force.

Corporate texts, such as transcripts of earnings calls, provide valuable information about how firms respond to current economic conditions. Economists can analyze earnings calls using a “text as data” approach to obtain real-time insight at the firm level, for example, to measure labor market tightness. In this blog post, we employ similar methods to explore firm-level AI-related chatter, sentiment and perceived risk.

We found a more than fivefold increase in sentences mentioning AI one year after the release of ChatGPT. These sentences were largely positive, with more than 30% being optimistic versus 10% pessimistic; the remaining sentences were neutral. However, firms expressed more uncertainty about AI with the introduction of ChatGPT.

Textual Analysis of Earnings Calls

Earnings calls are quarterly meetings where executives of publicly traded companies discuss the financials and current operations of their firms and factors that may impact future performance. For our analysis, we parsed transcripts of 185,999 earnings calls from 7,047 U.S. firms between the second quarter of 2008 and the first quarter of 2024; the transcripts were supplied by S&P Global.

To construct our measure of AI mentions, we began by identifying keywords related to AI and large language models, which are systems that uncover and use patterns from written texts to generate responses. Examples of the keywords include “machine learning,” “chatgpt” and “neural network.” We identified sentences containing these keywords as AI-related.

In addition, we used “interaction” words denoting positive, negative and risk sentiment. Our interaction words were repurposed from a sentiment lexiconThis dictionary is developed by Tim Loughran and Bill McDonald of the University of Notre Dame. designed to assess the sentiment of financial text.For details of our method, see this 2024 working paper, “Economic Surveillance Using Corporate Text,” by Tarek A. Hassan, Stephan Hollander, Aakash Kalyani, Laurence van Lent, Markus Schwedeler and Ahmed Tahoun. For example:

• “Disparage,” “dangers” and “failings” all have negative sentiment.
• “Efficiently,” “tremendous” and “advantageous” all have positive sentiment.
• “Hesitating,” “endanger” and “uncertainty” are all related to risk.

We constructed our sentiment measures by counting how many sentences in each transcript contained AI keywords along with the positive, negative or risk interaction words. For example, a sentence containing “machine learning” and “advantageous” would be counted as an AI-related sentence with positive sentiment.

Then, we computed net sentiment by subtracting negative sentences from positive sentences. To validate our measure, we performed a human audit by reading snippets of text containing sentences with both keyword and interaction words, as well as the sentences before and after (for additional context). This allowed us to confirm that our measure was successfully picking up AI chatter.

AI-Related Discussion Surged after the Release of ChatGPT

First, we look to see how AI chatter has changed over time. The figure below displays the number of sentences containing AI keywords per earnings call from the first quarter of 2012 until the first quarter of 2024. Until late 2016 there was very little AI-related chatter, with an average of about 0.02 sentences per earnings call.

Starting in early 2017, we see a steady, gradual increase in firms’ discussion of AI. Notably, we observe a large increase in AI chatter beginning in 2023. In fact, the chatter increased from 0.35 AI-related sentences per earnings call in the fourth quarter of 2022 to 2.17 in the third quarter of 2023. This suggests an increase in all types of AI-related discussion after the release of ChatGPT in November 2022.

Discussion of AI Rose Sharply in 2023

SOURCES: S&P Global and authors’ analysis.

NOTE: The red dashed line marks the fourth quarter of 2022, when ChatGPT was released.

Mostly Positive, and Some Negative, Sentiment about AI

In addition to looking at the overall increase in AI chatter, we wanted to know how firms are perceiving AI. The table below displays sample excerpts containing positive and negative AI sentiment from earnings calls. Frequently, optimistic firms cited increases in productivity and opportunities for growth. Firms also gave examples of new products delivered to market or increased efficiency with the assistance of artificial intelligence. Negative mentions included cybersecurity threats and copyright infringement.

The figure below displays the net sentiment—the difference in the fraction of positive and negative sentences about AI in earnings calls—over time. Starting in 2017 (only after which is there nontrivial AI chatter), firms overall were optimistic about AI. More than 30% of AI mentions in earnings calls had a positive sentiment, versus only about 10% with negative sentiment. Surprisingly, the release of ChatGPT had not significantly altered the net sentiment about AI among the firms.

Net Sentiment about AI Hasn’t Changed Much Following ChatGPT

SOURCES: S&P Global and authors’ calculations.

NOTE: The red dashed line marks the fourth quarter of 2022, when ChatGPT was released.

AI-Associated Risks on the Horizon

In addition to firms’ sentiment, we also wanted to know if firms are associating AI with potential risks and uncertainty. We constructed a measure of AI-related risk by dividing the number of sentences that contain both AI- and risk-related words by the total number of AI-related sentences in a transcript. The table below contains sentences flagged for AI-related risk.

Firms discussing potential risks of artificial intelligence cite similar concerns as those with negative AI sentiment: These include security risks and copyright violations. Other mentions of risk include misuse, unpredictability and a lack of understanding of AI’s capabilities.

The figure below shows the fraction of sentences about AI that also discuss risks. Before the introduction of ChatGPT, firms mentioned risks about AI only around 3% of the time. However, after the fourth quarter of 2022, this risk measure almost doubled to 6%. While the overall sentiment hadn’t changed after the release of ChatGPT, firms definitely expressed more uncertainty about how AI will shape the future.

Mentions of Risk Surrounding AI almost Doubled in Late 2023

SOURCES: S&P Global and authors’ calculations.

NOTE: The red dashed line marks the fourth quarter of 2022, when ChatGPT was released.

Conclusion

Utilizing textual analysis of earnings calls allows us to examine real-time AI-related discussion, sentiment and risk at the firm level. Our findings suggest that AI-related chatter increased over fivefold after the release of ChatGPT and was majorly positive in sentiment. Positive AI discussion included references to the launch of new products and increased efficiency and productivity, whereas negative mentions generally concerned cybersecurity and copyright risks. However, firms also expressed more uncertainty because of the introduction of powerful large language models. In our next blog post, we will discuss whether an increase in positive AI discussion translates to an increase in firm investment.

Notes

1. This dictionary is developed by Tim Loughran and Bill McDonald of the University of Notre Dame.
2. For details of our method, see this 2024 working paper, “Economic Surveillance Using Corporate Text,” by Tarek A. Hassan, Stephan Hollander, Aakash Kalyani, Laurence van Lent, Markus Schwedeler and Ahmed Tahoun.