The Rise of the Robotic Tax Analyst

Benjamin Alarie

Benjamin Alarie is the Osler Chair in Business Law at the University of Toronto and the CEO of Blue J Legal Inc.

In this article, Alarie, with the help of GPT-3, a large language model artificial intelligence system, reviews the predictions made in the Blue J Predicts column in 2022, and he speculates on AI’s role in tax research and analysis by 2030.

Copyright 2023 Benjamin Alarie.
All rights reserved.

I. Introduction

As a bold taxwriting experiment, this installment of Blue J Predicts has been generated with the help of an AI assistant, OpenAI’s “Generative Pre-Trained Transformer 3” (GPT-3).¹ GPT-3 is a large language model developed by OpenAI and backed by Microsoft. It is an inexhaustible generator of text and can write with accuracy in English about almost any topic. It has performed its duty, with my human companionship, indefatigably.

This isn’t the first time that a legal academic has invoked a robotic coauthor, and I expect that these kinds of tools will become increasingly commonplace.² I expect that eventually they will be about as remarkable as using a spelling or grammar checker. At this moment, however, before the rise of the robotic tax analyst, using GPT-3 to help write this article is likely to raise some eyebrows.

In October 2021 I produced a peer-reviewed law review article with my academic colleague, the late (and great) tax law professor Arthur Cockfield of Queen’s University. The article was notable for being the first peer-reviewed law review article to extensively leverage GPT-3 in its production.³ In that article — after a short introduction penned by us, Benjamin Alarie and Cockfield — we gave GPT-3 control of the metaphorical keyboard and allowed it to produce its textual analysis uninterrupted and unedited.

The results were mixed and intriguing, and certainly pointed in the direction of future possibility. In our view, GPT-3 had potential. In the article, we speculated on the future of AI in legal scholarship and asked provocatively in the title, “Will Machines Replace Us?”⁴ Cockfield and I concluded that “although GPT-3 is not up to the task of replacing law review authors currently, we are far less confident that GPT-5 or GPT-100 might not be up to the task in future.”⁵

Although these words were published just over a year ago, technological developments in 2022 suggest that we were probably somewhat conservative in our gauging of the likely rate of improvement of AI in the form of large language models. The rapid progress in the power of these models has been thoroughly impressive, with more progress to come. To help showcase the power of GPT-3 to readers of Tax Notes, I believe that I am the first author in these pages to rely on GPT-3’s help with drafting a contribution.

As 2023 begins, developments with large language models, most prominently ChatGPT, a recent offshoot of GPT-3, are generating buzz in legal technology circles. They are also triggering wariness among some in legal academia and practice. On November 30, 2022, OpenAI made ChatGPT available for public use through an open beta. In less than one week, the system was deployed by more than 1 million registered users. The extremely rapid adoption was driven by claims — and plenty of evidence — of impressive algorithmic feats performed by ChatGPT. The capabilities of ChatGPT reportedly include achieving passing scores on practice bar and medical board examinations.⁶

Within days of the release of ChatGPT, Andrew Perlman, the dean of law and a professor at Suffolk University, prompted the program to write an essay on its own likely influence in the future of law. The abstract to the essay, which Perlman assures us he wrote himself, concludes with the observation that “the disruptions from AI’s rapid development are no longer in the distant future. They have arrived, and this document offers a small taste of what lies ahead.”⁷

It’s not only GPT-3 and its progeny that have been making waves. Other algorithms garnered media attention in 2022 for exhibiting surprisingly strong image-generating capabilities. These latent text-to-image diffusion models, which include DALL-E 2, Stable Diffusion, and Midjourney, have been widely used, criticized, deployed, and celebrated.

The New York Times reported in September 2022 that an image generated by a text-to-image diffusion model won a digital art competition at the Colorado State Fair, causing a furor among artists, fellow competitors, and social media users.⁸ It is fair to say that machine learning and AI are beginning to make their influence felt in some unexpected places. There is little reason to expect these and related developments to abate at the doorstep of tax analysis.

To wrap up Blue J Predicts for 2022, we will do some Dickensian stocktaking, visiting Blue J predictions past, present, and future.⁹ The initial order of business is to retrospectively analyze how the Blue J Predicts analyses have fared in 2022. Have our predictions mapped on to reality? Have the cases gone the way the machine-learning models and the algorithms have anticipated? Broadly, the answer is yes, although with the important caveat that we do not yet have the final word from the courts in many of the cases.

We then consider the state of AI and machine learning in tax research and analysis. We outline some new ways in which machine-learning technology is poised to affect tax research and analysis this year, at Blue J and beyond. Finally, we look ahead to anticipate how AI could affect tax analysis and research, and explain why it is important to begin turning our attention to the possibilities.

II. Past Predictions: Blue J Predicts in 2022

Blue J Predicts routinely identifies pending or recently decided tax cases that are amenable to algorithmic analysis, to contribute to tax knowledge and to highlight the promise and power of Blue J’s machine-learning models. We published 10 of those analyses in 2022. In two of the 10 installments of Blue J Predicts, we did not make an explicit prediction about the most likely outcome of a tax dispute. On those occasions, it was usually because there was substantial uncertainty about the facts, significantly affecting the predictability of the result of the case. Blue J’s machine-learning models are designed to make accurate predictions based on assumed facts. If the set of facts is clear, Blue J can provide an accurate prediction.

This can be valuable. Tax practitioners often find it difficult to confidently provide authoritative advice in areas of tax law that lack significant guidance and can generate many — sometimes many hundreds of — litigated cases with seemingly contradictory and internally inconsistent outcomes. This can create confusion and uncertainty for taxpayers and their advisers. Indeed, for a determined tax analyst, it can be difficult to discern the significance of an aspect of a received legal test when studying a large body of cases alongside other primary and secondary sources of law.

Fortunately, the development of more powerful computing and machine learning can help to enhance the signal and reduce the noise, making it easier to comprehend and make accurate predictions of what tax law requires. That is precisely what Blue J does with its machine-learning models, achieving more than 90 percent accuracy in its predictions.

The table outlines the Blue J Predicts installments that appeared in 2022 and summarizes the key information about each prediction. In many of the cases, the predictive analyses are rather nuanced. If applicable, the predicted prevailing party reported in the table is based on the facts as determined at trial. When there was no clear predictive result on the base case facts, the predicted prevailing party field is incomplete.

In the remaining eight installments of Blue J Predicts, we predicted the most likely outcome of the tax dispute, based on the analysis performed by Blue J Tax. In six of those eight instances, Blue J predicted the outcome accurately, meaning that the court rendered the same decision as Blue J’s prediction. We are still awaiting the outcome of the litigation in the other two cases.

Importantly, the goal of Blue J Predicts isn’t necessarily to provide a precise prediction of the outcome of a tax case. Instead, it is intended to demonstrate the usefulness of Blue J Tax in better understanding and analyzing tax law to form better predictions and make more informed decisions. The accuracy of Blue J Predicts in 2022 can be seen as a testament to the promise and power of its machine-learning models.

We urge interested readers to refer to the original analysis for the complete treatment of several scenarios like the base case facts. Also, each original analysis will identify the limitations of each set of analyses and the conditions upon which the predictions are based.

III. State of Predictive Tax in 2023

Let us now turn our attention to the capabilities and implications of machine learning for tax research and analysis. There are four key areas in which AI and machine learning are going to make significant contributions in 2023: predictions, research recommendations, intelligent diagramming, and large language models. In 2023, and beyond, the increasing power of AI will continue to affect the practice of tax law in new and interesting ways. As with many other tasks that require significant amounts of data and analysis, the use of AI in the tax context will become more widespread.¹⁰

In general, an area of AI that is expected to affect the tax world in 2023 and beyond is natural language processing (NLP). Tax regulations are often dense and difficult to interpret. NLP systems are able to quickly process and parse these regulations, revealing nuances and subtleties in the language that might be overlooked by a manual review. While NLP systems are already being used to automate simple document reviews, more advanced tasks, such as summarization, keyword extraction, and automated risk scoring, are also becoming increasingly common.

Moreover, AI is proving useful in the area of dispute resolution. AI models are capable of automatically generating persuasive arguments using the same legal principles that attorneys use in arguing their cases. This has the potential to significantly reduce the time and costs associated with dispute resolution; it is debatable how much this will emerge in 2023. Time will tell.

A. Machine-Learning Models of Tax Issues

As highlighted in Blue J Predicts, Blue J’s predictive models are useful for tax practitioners. By leveraging the predictive power of machine learning, Blue J can accurately predict the outcomes of tax cases using various assumed fact scenarios. The models are built with massive amounts of data collected from tax cases and legal sources and are trained to make accurate predictions based on the facts of each case. The models are consistently being refined, updated, and expanded as new examples of tax disputes become available. The resulting predictions are highly accurate and reliable, demonstrated by the more than 90 percent accuracy generated by Blue J in its analyses.

B. Research Recommendation Engine

Blue J’s research sources recommendation engine is a powerful new tool that can be used to find the most relevant and reliable sources of law and administrative guidance for any tax case. By leveraging the power of AI and machine learning, the engine can quickly identify and recommend research sourced, including tax statutes and court decisions, based on the work that a tax researcher is doing in Blue J Tax. The engine considers the tax context of each research task and uses this information to provide more precise and accurate recommendations. As the engine continues to learn, its recommendations will become even more precise and effective.

C. Intelligent Diagramming

With the growing complexity of tax law and the commensurate complexity of corporate transactions, practitioners often find it cumbersome to accurately depict and rapidly revise many stepped transactions using off-the-shelf office presentation software. The existing tools to aid in diagramming transactions are limited to producing static diagrams, which often fail to provide enough context, require time and diligence to update, and bring no extra insight on how to optimize the tax consequences.

That is where Blue J’s intelligent diagramming and knowledge graphs come in. By leveraging the power of machine learning and NLP, the platform can help users build detailed diagrams of the most complex transactions and generate clear visual representations of their connections and implications. Those diagrams not only provide a more comprehensive and intuitive understanding of tax law but also enable faster and more accurate analysis of the relevant facts and applicable rules when paired with the research sources recommendation engine.

D. Large Language Models

As mentioned previously, the existing tools for legal research and analysis are often limited by their reliance on static diagrams and charts. However, one of the most promising types of NLP models is large language models. By leveraging powerful neural networks, these models can generate comprehensive yet understandable summaries of legal documents and sources. This enables practitioners to quickly identify key concepts and principles in a source, coming to a better understanding of its implications. The promise of large language models is tremendous. As these systems become more powerful and widely applied, legal researchers and practitioners will be able to conduct more efficient and effective analysis of tax law.

E. Summary of What’s to Come in 2023

Legal research tools that use AI-driven methods, such as Blue J’s predictors and its new research sources recommendation engine, are becoming increasingly popular for their ability to quickly locate and curate the most relevant and up-to-date legal sources for any tax issue.

In addition to directing experts to the most relevant guidance resources, AI-driven tools can accurately generate detailed diagrams of complex transactions, highlighting connections and implications that could be overlooked. The use of NLP and large language models will increasingly enable research platforms to quickly scan and extract key concepts from various sources, helping practitioners better understand their implications — both their potential pitfalls as well as opportunities for improved planning outcomes. Machine-learning models enable practitioners to predict dispute resolution outcomes more accurately by testing various scenarios based on different sets of assumed facts. The near term of tax research and analysis will continue to witness the use of AI-driven technologies to enable significant improvements.

IV. Role of AI in Tax Analysis in 2030

As AI improves, the field of tax analysis will likely evolve and by 2030 be well on its way to being fundamentally different. Tax practitioners are already adding value to their services by leveraging these technologies to bring greater precision and accuracy to their work, as well as by providing clients with more comprehensive advice. Moreover, governments will continue to use AI to detect and punish tax evaders and make sure proper collections are made. Indeed, AI will likely become a more central and necessary part of competent tax advice. AI systems, especially when combined with advances in NLP and large language models, will be able to accurately predict the likelihood of success at the planning stages of a transaction, well before a party embarks on a case, helping practitioners to make better-informed decisions. AI will also be used to identify patterns or risks that may not have been noticed before.

In the interest of being provocative, here are five predictions about how AI and machine learning will increasingly affect tax research and analysis over the rest of the 2020s.

Prediction 1: Governments will use AI to combat aggressive tax planning that can be difficult to detect since it often involves complex transactions that are difficult for humans to process. AI algorithms, however, will be able to quickly detect complex patterns of tax planning that could indicate a violation, making it easier for governments to identify and prosecute offenders. Also, AI can be used to find errors and suggest corrections to ensure compliance with the applicable tax laws.

Prediction 2: AI will become an even more essential tool to detect and prosecute tax evaders. Tax evasion schemes, particularly those involving money laundering or offshore accounts, can prove difficult to detect because of their complexity and the numerous jurisdictions involved. However, as AI continues to improve, governments can leverage automated processes to detect and investigate offenders and combat tax evasion at unprecedented levels. AI technology can be used to automatically find patterns typical of tax evasion, which can then be used to identify and investigate offenders.

Prediction 3: AI will be used to predict the outcome of tax disputes with high accuracy. As AI technology continues to improve, tax disputes will become easier to resolve and the accuracy of dispute resolution will increase. AI algorithms can analyze large bodies of tax case law and primary and secondary sources of law, as well as other external data sources, to identify predictive trends that could help to make accurate predictions regarding the outcomes of cases. This will enable tax practitioners to make better-informed decisions earlier in the dispute and better anticipate the potential outcomes of cases.

Prediction 4: AI will allow for more natural and comprehensive legal analysis. As AI technologies continue to improve, NLP algorithms will be able to process large bodies of legal text more quickly and accurately than ever before. This will allow tax practitioners to quickly and accurately identify the pertinent points in a text and generate conclusions and predictions that are more representative of the content in the legal document.

Prediction 5: AI-driven analysis will become an essential part of competent tax advice. AI-driven analysis capabilities are rapidly becoming essential tools for legal practitioners, and they will become even more important and necessary in the coming years. As AI improves, it will become increasingly difficult to do legal analysis without it, as the sophistication of legal cases and the amount of data involved continues to grow. AI algorithms will also be used to proactively identify potential risks and alleviate them before they become disputes. AI can be used to detect errors or deviations from applicable tax laws, as well as detect potential opportunities that could be taken to ensure the most favorable tax treatment. By leveraging AI technology to identify and resolve potential issues before they become disputes, businesses and individuals can save time, effort, and money.

V. Conclusion

It is exciting to consider the possibilities that AI and machine learning are creating within tax practice and the related professions. Our own success in predicting outcomes in Tax Notes in 2022 indicates that AI technology promises to continue to have an effect.

As we look ahead to the rest of 2023 and beyond, it is reasonable to assume that AI will continue to make advances in tax practice and other related fields. Distilling the above discussion of AI technology’s present and future, two areas look particularly promising for new developments. These are the automation of legal research and the application of knowledge graphs. Automation of legal research is something that has been talked about for years and is only now starting to become a reality. It is likely to transform the way that legal services are provided, leading to greater efficiency and cost savings. The second promising development, knowledge graphs, is already being used in tax practice to represent legal relationships and generate insights. These knowledge graphs can provide a basis for finding the best research resources and for reasoning on complex legal issues.

In conclusion, the practice of tax is likely to be increasingly affected by AI and related technologies in the coming years. As AI and machine-learning models become increasingly proficient, they will play an ever-larger part in tax planning and analysis, dispute resolution, and other areas of tax practice. It remains to be seen just how far AI will progress in the tax field, what new tax planning opportunities will arise, and which areas of tax practice will be most affected. But it will be an interesting journey, and one that is likely to profoundly shape the field of tax in ways that are challenging to anticipate.