IRS enforcement is changing with artificial intelligence. To avoid discrimination and bias in tax administration, observers say transparency and oversight is crucial, especially when it comes to the data used in AI models.
AI is a tool with the potential to make tax administration more effective, efficient, accurate, and fair, said Cary Coglianese of the University of Pennsylvania Carey Law School. But if the AI is built into an automated system that isn’t carefully deployed, it can create risks and inaccuracies, he said.
Government agencies — including the IRS — should consider risks for bias in the data that algorithms train on, which can get replicated into what the algorithms are doing, Coglianese said.
The IRS announced that it is using AI to select large partnerships for audit and said it is piloting the use of AI to select earned income tax credit recipients for audit. But the agency has made little information available to the public regarding its use of AI and has declined to release the data it is using to test and train algorithms that are used to select large partnerships and other tax returns for audit.
Tax Analysts requested testing and training data for the algorithms used for audit selection in campaigns in the IRS’s Large Business and International Division, including those targeting inflated cost of goods sold, partnership losses in excess of partners’ basis, sales of partnership interest, and high-income nonfilers.
The IRS denied Tax Analysts’ Freedom of Information Act request for that information, citing FOIA exemption (b)(7)(E), which allows agencies to withhold records if their release would disclose techniques and procedures for law enforcement investigations or prosecutions.
Tax Analysts appealed the decision, asking the IRS to instead provide documents with redactions. The IRS Independent Office of Appeals determined that the request was denied using the correct FOIA exemption.
‘Tight-Lipped’
Robert J. Kovacev of Miller & Chevalier Chtd. said the IRS is “very tight-lipped about exactly what they’re doing” when it comes to using AI.
“They don’t want to give away their sources and methods, because if that happened then people would just plan around whatever the algorithm happens to be,” said Kovacev, who previously worked as an attorney in the Justice Department Tax Division.
“You don’t want to have people reverse engineering, giving them a roadmap for noncompliance,” said James R. McTigue Jr. of the Government Accountability Office.
Still, Nina Olson of the Center for Taxpayer Rights said it is important to know the data sources that the IRS is using to train its algorithms because some may be unreliable and unrefined. She suggested that the agency release explanatory information about the data to the public.
“We don’t know the databases that they’re using. Are they being trained on IRS data? Are you pulling together external data and putting IRS data in that? How is it being used? What is it being used for?” said Olson, who is a former national taxpayer advocate and is a member of Tax Analysts’ board of directors.
The IRS has “volumes and volumes of data — petabytes, hundreds of petabytes of data” — internally, IRS Chief Technology Officer Kaschit Pandya said September 10 at a Tax Executives Institute event. One petabyte is 1,000 times larger than a terabyte and 1 million times larger than a gigabyte.
The agency is already using the data to develop tools that employees can use to conduct and document interactions with taxpayers and has been using it in machine learning applications for decades, Pandya said.
Pandya acknowledged the potential problems associated with using AI. “We know that along with AI and all the wonderful things that it could possibly or potentially do, there are some known dangers of it,” he said. Those dangers include bias and hallucinations — instances in which an AI algorithm creates false or nonsensical information.
More Data, More Problems?
Olson said some of the data sources may be developed for a purpose other than what the IRS uses it for, which can create problems because the data is being interpreted in a way that was never intended.
For instance, Olson said that Congress authorized the IRS to assess tax using math error procedures when a taxpayer claims the EITC for a child who is shown on the federal case registry of child support orders as being in someone else’s custody. After a study found that only about 39 percent of the returns selected based on data mismatches were correctly selected, the IRS agreed not to assess math errors based on the data, according to Olson’s 2015 annual report to Congress.
“I really worry that if we can’t see what the databases are that machines are being trained on, then that will lead to inaccurate results, which will harm taxpayers and waste IRS resources,” Olson said.
A 2019 report from the European Union Agency for Fundamental Rights said that “AI systems based on incomplete or biased data can lead to inaccurate outcomes that infringe on people’s fundamental rights, including discrimination. Being transparent about which data are used in AI systems helps to prevent possible rights violations.” Assessing data can help ensure it is high quality, the report said.
Concerns about data quality surfaced in 2023 after Stanford University’s Institute for Economic Policy Research found that Black taxpayers claiming the EITC are about three to five times more likely to be selected for audit by the IRS than non-Black taxpayers because of the algorithms it uses for audit selection.
In response, IRS Commissioner Daniel Werfel said the agency would reduce its usual volume of EITC audits and that it was testing new tools to reduce racial disparities in audits.
Pete Sepp of the National Taxpayers Union said releasing information about the data used in the agency’s AI models wouldn’t necessarily “open up a gateway for tax cheats to try and beat whatever the IRS intends to use AI for. Rather, it’s simply to have an evaluation in the way that many government agencies have been subjected to when they’re implementing AI.”
Releasing that type of information “might actually serve the cause of enforcement and compliance,” Sepp said.
Models
In a July 2023 report, the GAO reviewed the IRS’s process for selecting partnerships for audit and found that its statistical models for selecting returns for audit were flawed because they were “developed without using representative samples of returns and with untested assumptions.”
The GAO suggested that the IRS address the modeling issues to better identify and audit partnerships that aren’t compliant.
The GAO’s recommendations for the IRS included using a representative sampling of partnership returns “to help identify additional noncompliance that may not be detected and to improve the agency’s understanding of the models’ effectiveness.” It also recommended that the IRS test and validate the “key assumptions used in IRS’s partnership models through analysis of data on audit outcomes or other research and develop a formal process for using audit results and other data as they become available to improve model performance.”
The IRS agreed with those recommendations and is working to address them, according to the GAO’s website.
Kovacev noted that the IRS isn’t required to tell taxpayers they have been selected for audit because of its use of AI as opposed to any other method. While there’s no way to know for sure if AI was used to select taxpayers for audit, Kovacev said he sometimes thinks a client was selected because of AI. The IRS should let taxpayers know when it uses AI, which can be done without revealing too much information, he said.
Kovacev said that when using AI, the IRS should make sure an agency official reviews what the algorithm is doing.
At a June 27 conference hosted by the New York University School of Professional Studies, LB&I Commissioner Holly Paz assured tax professionals that there is “a tax technical person at every stage” in the process of using AI and machine learning to select large partnerships for examination.
“Work is not being selected by a machine without any kind of human intervention by someone who knows what they’re doing,” Paz said, but she didn’t say whether that person is an IRS employee or a contractor, or what their position at the agency is.
“It’s important for the IRS to be much more transparent with the public about exactly what they’re doing and what safeguards they are putting in place when they’re using these technological tools,” Kovacev said.
An IRS spokesperson said the agency wouldn’t elaborate further on its use of AI to select large partnerships for audit.
More Transparency? More Scrutiny?
Reza Rashidi of the IRS Office of Research, Applied Analytics, and Statistics told Tax Notes June 13 that the agency hadn’t thought about providing data cards, which provide a description of data used in AI models that is similar to a drug fact label, or other information that has been redacted to remove or hide sensitive information about the data being used.
Rashidi said the IRS might be able to release some explanatory material that provides nonsensitive information about what data is being used and how. The IRS declined to provide additional information or make Rashidi available for a follow-up interview.
McTigue said he would welcome the explanatory material. “From a transparency standpoint . . . that would be helpful,” he said.
Olson said there are people in Europe who work on transparency in AI because their laws require it. “That’s what we have to get to in the United States,” she said.
“They’re dealing with the same thing — the fear of reverse engineering. They’re just not having that be the end of the conversation,” Olson said. “They’re working with the data scientists, the mathematicians, the engineers, and the ethicists and the lawyers and the economists to figure out what they can explain.”
Olson said if the IRS is going to use AI, there should be public notice and comment.
If “this is the system that we’re doing, and we’re going to use this data . . . then people should be able to say, ‘I don’t think that this data is accurate,’” Olson said.
Taxpayers can challenge a decision if they know what the issue is, but they can’t if it’s a "black box,” Olson said.
Coglianese, who wrote a statement for the Administrative Conference of the United States (ACUS) that provided a framework for governmental use of machine learning, said AI systems should be scrutinized to ensure they’re accurate. Coglianese suggested the IRS could hire independent third-party verifiers or auditors to review the agency’s systems.
In a recent article in Tax Notes, Jin Soo Lee and Rami Khoury of Taylor Nelson Amitrano LLP proposed creating a “data integrity and ethics lab” (DIEL) within the IRS to evaluate training data for bias and unlawful discrimination.
“The DIEL would be responsible for reviewing machine learning models and data analytics algorithms during planning, development, and deployment to ensure compliance with legal and ethical standards,” the authors said.
Employee Guidance
ACUS released recommendations on AI in 2020 that say agencies should “consider whether they have, or can obtain, data that appropriately reflect conditions similar to the ones the agencies’ AI systems will address in practice; whether the agencies have the resources to render the data into a format that can be used by the agencies’ AI systems; and how the agencies will maintain the data and link them to their AI systems without compromising security or privacy.”
“Agencies should also review and consider statutes and regulations that impact their uses of AI as a potential collector and consumer of data,” ACUS said.
Regarding transparency, ACUS said that in contexts such as enforcement, "agencies’ legitimate interests in preventing gaming or adversarial learning by regulated parties could militate against providing too much information (or specific types of information) to the public about AI systems’ processes.”
In October 2023 President Biden issued an executive order on the safe use of AI that encourages independent regulatory agencies to emphasize or clarify “requirements and expectations related to the transparency of AI models and regulated entities’ ability to explain their use of AI models.”
The IRS released interim guidance in May with AI governance and principles. Sepp said the guidance falls short in certain areas — for example, it says the chief data and analytics officer also serves as the IRS official responsible for overseeing the agency’s AI governance program.
Decisions interpreting and implementing the directives and exceptions in the interim guidance for public release of AI information are within the sole discretion of Treasury and IRS officials.
“The Service is in the middle of one of the biggest, most complex technology transformations in its history. Here’s hoping that the [chief data and analytics officer and responsible AI official] has a lot of help with this,” Sepp said. “Otherwise, we may very well find ourselves in a situation a year from now, two years from now, where the promised continual oversight of AI implementation has slipped.”
The interim guidance directs the IRS to publicly release AI code and models that are in active use, but it excepts broad categories — contractual obligations to contractors and other third parties, for instance. It also provides for full or partial disclosure of the data the IRS uses, subject to privacy considerations and section 6103, which prohibits disclosure of tax returns and return information.
The guidance also indicates that human review will be required at least annually when the IRS is using “new or existing covered safety-impacting or rights-impacting AI,” which doesn’t seem frequent enough, according to Sepp.
“Sooner or later, Congress is going to have to explore some additional guardrails around AI’s use in government, and especially with the IRS — no other agency interacts with as many people more frequently than the Service — and if there’s anywhere in the federal government that AI adoption has to proceed slowly and thoughtfully, it’s here,” Sepp said.