This research note was originally published by the FactSet Blog
Natural language processing (NLP) and machine learning (ML) are approaches that are becoming increasingly popular tools used to add value in the investment process. These approaches give us the ability to analyze unstructured data such as news, corporate filings, social media, and other sources to derive meaningful content.
Recently, this same technology has been used in an application that has historically been the job of thousands of analysts over countless hours around the globe: analyzing corporate earnings calls. Drawing from FactSet’s dataset of over 1.7 million corporate documents, we apply and compare three popular NLP approaches: FinBERT, Loughran McDonald (LM), and Alexandria Technology to identify any agreement among them and to compare their historical performance.
Extracting Facts, Figures, and Sentiment from Earnings Calls
Earnings calls have always been a source of integral information for investors. During these calls, companies discuss financial results, operations, advancements, and hardships that will steer the course and future of the company. This information is a key resource for analysts to forecast the ongoing and future financial performance of the company. The calls are split up between Management Discussion (MD) where executives present financial results and forecasts, followed by Questions and Answers (Q&A) where participants such as investors and analysts can ask questions regarding the MD results.
Using NLP and ML, we can analyze these calls in near real time. By parsing and scoring them for sentiment, we have an overall view of thousands of calls as well as a topic-by-topic (or sentence-by-sentence) detailed view of what was said, who said it, and what the sentiment was without ever having to dial in or read through the transcripts. Armed with the quantitative information derived from these calls, we can then explore various ways to incorporate and enhance our investment processes.
Previous research shows that using NLP for earnings calls can add orthogonal alpha not explained by traditional risk and return factors, but the question becomes, which method of NLP we should be using to analyze this data, and does it make a difference? To answer this question, we take a look at three NLP approaches and use them to apply sentiment to transcripts sourced from FactSet’s document distributor.
Machine Learning vs. FinBERT vs. Loughran McDonald
The first NLP model is the lexicon or “bag of words” approach of Loughran McDonald that uses a dictionary of words or phrases that are labeled with sentiment. The second is the popular machine learning model for finance FinBERT, based on the Google BERT language model but trained specifically for financial contexts. Finally, we look at Alexandria’s machine learning approach which is uniquely trained from analyst earnings call labeling. We apply each model to our sample universe of the S&P 500 for a period between 2010-2021 to gather sentiment scores at the individual sentence/topic level and the aggregate security level.
For the topic level, we found very low correlations between Alexandria and both LM and FinBERT (0.14 and 0.17 respectively), and the highest correlation to be between LM and FinBERT at 0.38. Once aggregated to the security level, we found slightly higher correlations between the three, as summarized in the second table below. We observe that despite all being NLP models specifically for the financial domain, there is large disagreement among them, particularly at the topic and sentence level...
REQUEST HERE TO READ THE REST OF THE FULL RESEARCH PAPER