21.10.1. Embedded Quality (EQ) Program for Accounts Management, Campus Collection, Campus Examination, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS), and Electronic Products and Services Support

(1) The Headquarter Policy divisions issue program goals based on the Balanced Measures.

(2) Policy reviews Quality Assurance programs as part of periodic reviews of Accounts Management, Compliance, Electronic Products and Services Support, Return Integrity and Compliance Services - Integrity and Verification Operations and Field Assistance programs.

(1) The mission of CQ&TS and EQ&TS is to provide unbiased, cross-Policy administration of the Embedded Quality Review System. The functions span both Campus Collection and Campus Examination and have an enterprise objective to measure performance. By design it is independent from other functions to ensure impartiality and maintain data integrity. CQ&TS and EQ&TS regularly interacts with the headquarters’ business owners and all campus/remote office operations to work quality issues including:

• Providing oversight and coordination of the cross-functional Embedded Quality (EQ) Program.

• Monitoring EQ program adherence including proper coding of reviews and all aspects of the sampling process.

• Sponsoring on-going cross-campus discussions to identify trends and issuing guidance and direction to correct global problems.

• Maintaining the EQ system which includes establishing workarounds for problems identified through coding.

• Updating EQ tools including job aids, system fields, attribute definitions, and monitoring and resolving Contact Recording issues.

• Establishing regulations and procedures and providing support for attribute coding for Campus Collection and Campus Examination reviews.

• Making all final-authority determinations on elevated rebuttals or adjustments.

• Serving as the Campus Collection and Campus Examination liaison to Accounts Management including coordinating system adjustments to ensure uniform and cohesive data collection.

• Sponsoring and supporting new Campus Collection and Campus Examination product lines for the system.

• Supporting coding consistency exercises with Campus Operations to promote consistent coding by NQRS and EQRS reviewers.

(1) The EQ National Support Staff is part of the Joint Operations Center (JOC) and provides support to all EQRS and NQRS users in all functions at all levels. The staff is responsible for:

• Serving as liaison between JOC and various Business Units in the Enterprise on program and quality issues

• Coordinating with Statistics of Income (SOI) on weighted report calculations and related programming

• Supporting business quality training initiatives

• Working with business units to develop and define new (Specialized Product Review Groups (SPRGs)

• Working with the business units to develop new reports as needed

• Working with various business Product Line Analysts to maintain Data Collection Instruments (DCIs) for current SPRGs

• Supporting the EQ Summits between business units and NTEU required in the contract

• Processing Reports Request Central (RRC) submissions for EQRS/NQRS data

• Issuing system alerts to customer base

• Ensuring official Quality measures (Customer Accuracy, Timeliness, Professionalism) as reported by the application are timely and accurate

• Ensuring managerial performance reviews as reported by the application are accurate and reliable

• Creating/submitting annual EQRS-C (Campus) enhancement UWR

• Creating/submitting annual EQRS-C Maintenance UWR and other UWRs as needed

• Working with IT (Applications Development, Embedded Quality Section) on all EQRS-C system-related issues

• Coordinating with IT in completing annual UWR submissions

• Maintaining EQRS-C database reference tables via OL-Matrix web application

• Serving as Application Point of Contact for all FISMA-related activities and requirements

• Maintaining Embedded Quality SharePoint site https://irsgov.sharepoint.com/sites/eqrsc-support

• Serving as IRM author/content owner of IRM 21.10.1, Embedded Quality (EQ) Program for Accounts Management, Campus Collection, Campus Examination, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS), and Electronic Products and Services Support.

• Processing BEARS requests submitted to the following entitlements: EQRS (CAMPUS)-HQ (EQRS-HQ), NQRS (NQRS (CAMPUS)) and PROD ELEV EQRS USER (EQRS)

(1) The P&A Chiefs for Accounts Management and Compliance are responsible for the Site Level Business Plans.

(2) The P&A Chiefs are responsible for site reviews of product lines in Accounts Management that provide data for the Balanced Measures.

(3) The operations managers are responsible for evaluative reviews on their employees separate and apart from the National Review process on the EQRS side of the database.

(4) If feasible, the Embedded Quality Program Manager and Process Improvement Specialist/Field Improvement Specialist should report directly to the P&A Chief. This eliminates potential conflicts of interest which may occur when more than one department manager is accountable for a product or service. If a formal position is not designated within P&A the responsibilities should be designated to one or more individuals in the operation to ensure that the obligation is met.

(1) This subsection only applies to remote and campus locations with functional areas where there is an existing QAM position. In offices/functional areas where no position exists, or in Campus Examination, management will need to ensure these duties are appropriately addressed.

(2) The QAM is responsible for the overall planning, administration, and evaluation of the quality-related sections of the Site Level Business Plans. The QAM will identify problems and work with management to solve them.

(3) The Site Level Business Plans outline procedures for the review of all areas of responsibility. This review process, when combined with CQRS/PAS data and other functional data, will help with evaluating the overall quality of operations and making recommendations for improvement.

(4) The QAM will serve as the Quality (QA) manager for the Operation, ensuring that designated quality resources are used to focus on quality improvement efforts.

(5) By using trend analysis, the QAM will determine the causes that adversely affect quality. The QAM will assist the management team in initiating processes for employees to improve their quality of service. It is important that lines of communication remain open among the QAM, the QR team, and management to identify problem areas, take appropriate corrective actions, and re-evaluate quality to ensure corrective actions result in improved quality.

(6) The QAM or other designated person within the operation will log in and date the completed review records that require rework, including reviews by CQRS/PAS, where defects have been identified including when FLASH was annotated in the Feedback Summary Remarks field. Managers must return corrected work to the QAM within five working days of receipt. See IRM 21.10.1.8.7(3), for additional instructions on FLASH remarks. The QAM will enter the completion date in the log. The QAM will monitor corrected work to ensure the timeliness and quality of responses to taxpayers.

(7) The QAM will act as the liaison between the CQRS/PAS staff and management and is responsible for communicating quality information to all managers in the operation.

(8) The QAM will identify training needs and recommend to the Training Coordinator and/or management, the type of training needed (e.g., on-the-job training or classroom instruction), and assist in the development of additional training exercises and workshops to meet those needs.

(9) The QAM and the CQRS/PAS staff will be responsible for the protection of NQRS DCIs and any supporting documentation from legacy systems. All documents and information (including taxpayer information) seen, heard, or handled must remain secure and confidential.

(10) The QAM will serve as the Embedded Quality liaison with managers.

(11) The QAM will be responsible for working with the Training Coordinator and/or management to ensure that EQ training is made available to all who need it. The QAM is responsible for providing technical training to the quality staff.

(12) The Field Improvement Specialists (FIS) will work closely with Headquarters and the PICA team to manage the portfolio of national projects and best practice evaluation. The expectations for the FIS positions include, but are not limited to, the following.

1. A regular solicitation for projects will be made to ensure there is no overlap between various sites or functions.

2. Field Directors recommending projects would oversee those selected.

3. Each Field Director will conduct at least one project with national impact each year.

4. Field Directors will have the latitude to use FIS resources for local projects.

5. All projects will be linked to the organization’s strategic goals and direction.

(1) Throughout this IRM section, the term "Quality Managers" will be used to include the duties of CQRS/PAS Managers, Embedded Quality Program Managers or other functional Quality Review Managers who perform site reviews for the national quality measure and/or local reviews for quality improvement.

(2) Quality Managers are responsible for ensuring the completion of the national and local reviews.

(3) Campus Collection and Campus Examination PAS Managers use the Campus Quality Review SharePoint Site (CQRSS) to assign and monitor the national reviews.

(4) Quality Managers ensure that all applicable work is sampled and reviewed within sample plan guidelines.

(5) The Quality Manager maintains the integrity and quality of the review system by monitoring and reviewing the quality analysts and clerical support. The Quality Manager should routinely monitor the quality sampling selection process. Refer to descriptions of the product lines later in this section for more information on sample universes and sampling guidelines. The Quality Manager should also perform a periodic (at least quarterly) spot check of the work to confirm the case volume matches the count provided by the operations. For example: verify the review content of a single folder to ensure that the volume of the work matches the number of cases in the folder. If these figures do not match, contact the operation to address the reason(s) for the discrepancy and to discuss ways to improve the process accuracy.

(6) The Quality Manager and staff determine the causes that adversely affect quality by using trend analysis to identify the most frequently made defects and root causes. The Quality Manager recommends corrective action and/or improvements to functional areas.

(7) The Quality Manager identifies sample cases that require rework (including cases where FLASH was entered in the Feedback Summary Remarks field) and may forward them to the appropriate functional area. This includes reviews by CQRS and CER where defects have been identified. All defects must be reworked to ensure each taxpayer receives the correct information and all appropriate adjustments are made to their account(s). The manager of the employee who created the defect is responsible for ensuring the defect is corrected within the applicable timeframe after the two-day consistency period. Flash defects must be corrected by the next business day. All other corrective actions not marked as Flash defects must be corrected within five business days.

(8) The Quality Manager is responsible for working with the Training Coordinator to ensure that the CQRS/PAS staff has the appropriate training.

(1) A quality analyst, clerk, or operation designee is responsible for pulling the daily sample.

(2) The quality analyst should perform an unbiased, consistent, and accurate review of all work.

(3) The quality analyst provides the QAM or PAS Manager with:

• Any cases identified as Flash for rework

• Analysis of types of errors identified during review.

(4) The quality analyst should provide the QAM, CQ&TS, EQ&TS, Embedded Quality Program Manager, PICA Product Line Quality Analyst, or National Support Staff, as applicable, with recommendations for corrections/improvements on:

• IRM 21.10.1, Quality Assurance - Embedded Quality (EQ) Program for Accounts Management, Compliance, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS) - Integrity and Verification Operations and Electronic Products and Services Support

• Embedded Quality Review Training Material

• The NQRS DCI

(5) The quality analyst will:

1. Code the appropriate attribute for each action. While considering the root cause of any defects, a single action should not be coded "N" in multiple attributes to avoid a trickle-down effect.

2. Review work using valid sampling techniques as approved by SOI for all SPRGs with sampling plans designed by SOI. See IRM 21.10.1.2, The Quality Review Process, to determine if the SPRG has a sample designed by SOI.

3. Record complete review results using the NQRS DCI.

4. Measure performance against established quality standards in the functional IRM, publications, and other approved reference sources.

(6) The quality analyst reviewing the work will need to complete training that is appropriate for the SPRG they will review. The following is a list of suggested training topics:

• Embedded Quality Review Training available on Integrated Talent Management (ITM)

• Quality Review command codes

• Automatic Data Processing (ADP)/Integrated Data Retrieval System (IDRS) training modules, including on-line accounts and adjustments, and Accounts Management and Compliance Services

• Coding Consistency Course

• Technical tax law (as appropriate)

• Interactive Tax Law Assistant (ITLA)

• Integrated Automation Technologies (IAT) Job Aids

• Accounts Collection Service Guide

• Correspondence guidelines

• Taxpayer Advocate Service guidelines and criteria

• Oral statement guidelines

• Procedural guidelines

• Communication skills

• Probing skills

• Timeliness guidelines

• e-help

• TE/GE Specialized Systems

• TE/GE Probe and Response Guide

• Continuing Professional Education (CPE) classes related to their assigned SPRG when available

(1) The Quality Analysts (PAS, QAC) is responsible for:

1. Pulling the daily sample (for TPR, AMSS, Adjustments. Campus Support, Non-CII, Spot-Checking Work Initiative, etc.)

2. Performing consistent, unbiased and accurate review of all work

3. Providing Managers with any cases identified as FLASH for rework and an analysis of the types of errors identified during review

4. Identifying IRM updates for Weekly Meetings and local training initiatives

5. Providing feedback for corrections/improvements to the tax analysts

(2) The local Quality Analyst will use the National Quality Review System (NQRS) Data Collection Instrument (DCI) to:

1. Code the appropriate attribute for each action

2. Consider root cause of each defect as a single action

3. Not code "N" in multiple attributes to avoid a trickle-down effect

4. Review work using valid sampling techniques and approved by SOI

5. Record complete review results using the NQRS DCI

6. Measure performance against established IRM procedures, publications, form instructions and other approved reference material.

(3) Identify and coordinate IRM procedures impacting quality. Follow IRM 21.10.1.2.7.1, AM IRM Procedural Update/SERP Alert Filtering and Distribution.

(4) Work with local PI Specialist (or PICA) to gather facts/data to justify procedural, systemic and program changes to improve work practices, policies and procedures. Forward sound recommendations to PPM, IPSO, IDTVA and RICS - i.e., Taxpayer Protection Program with solid data to justify changes.

(5) The Quality Analyst reviewing the work will need to complete Training lessons that are appropriate for the Specialized Product Review Group (SPRG) they review. The following is a list of suggested training topics:

• Embedded Quality Review Training

• Quality Review command codes

• Automatic Data Processing (ADP)/Integrated Data Retrieval System (IDRS) training modules, including on-line accounts and adjustments, and Accounts Management and Compliance Services

• EQRS Coding Consistency Training Courses (3 courses)

• Technical Phone Courses (as appropriate)

• Technical Paper Courses (as appropriate)

• Technical Probe and Response Guide

• Interactive Tax Law Assistant (ITLA)

• Accounts Collection Service Guide

• Correspondence guidelines

• Taxpayer Advocate Service guidelines and criteria

• Oral statement guidelines

• Procedural guidelines

• Communication skills

• Probing skills

• Timeliness guidelines

• e-help

• TE/GE Specialized Systems

• TE/GE Probe and Response Guide

(6) The local Quality Analyst will conduct "SPOT-CHECKS" of Adjustments Paper closures to stop incorrect account adjustments, incorrect letters (or no letter when required) and erroneous refunds. This task may be designated to another skilled employee with the appropriate skills to conduct the spot-check reviews (e.g., DMFC skills, Leads, CER Specialist).

1. Each day search CII prior day closings by organization code and attempt to select 100 cases per day (e.g., 25 cases for 4 teams = 100 cases a day).

2. Procedural errors maybe skipped (unless they are egregious).

3. Long or complex cases maybe skipped.

4. On the Adjustment Spot Checking Spreadsheet record the actions and statistical information required.

5. Forward the Spreadsheet to Process Improvement (PI) Specialist by Tuesday of the following week.

6. The manager is to return the case to the CSR for immediate corrective action.

7. Each Manager is responsible for ensuring the case is closed correctly.

8. Manager and/or Lead is to share constructive feedback with employees to eliminate future mistakes. The case is non-evaluative.

9. Per IRM 21.5.1.5.1(17), CII General Guidelines, close as MISC. Close as MISC cases that are reopened from archives or reassigned to the CSR to correct a mistake.

   Note: Using the “Close as MISC” button bypasses the National Program Quality Review (CQRS).

(1) Centralized Evaluative Review (CER) Technicians and Specialists are quality reviewers on the Planning and Analysis staff at Accounts Management campus and remote sites. The CER review teams are supported by a CER Manager and CER Lead. The CER teams report to the Embedded Quality Program Manager.

(2) CER Technicians and Specialists duties are:

• Maintain research and technical skills for all current IRS work related procedures

• Within three (3) business days of assignment, perform reviews of toll-free phone calls and/or Correspondence Imaging Inventory(CII) casework using the Embedded Quality Review System (EQRS) database

  Note: CER Technicians will review CSRs grade 05 through 07 only. CER Specialists will review all CSRs and TLS/TSS employees assigned.

• Adhere to disclosure procedures to maintain confidentiality

• Generate reports from EQRS to identify common errors and analyze trends

• Prepare and present coding consistency cases for coding consistency meetings

• Develop and share with site stakeholders:

1. Job aids and training documents based on deficiencies identified through EQRS reports

2. Quality improvement initiatives with recommendations for addressing quality deficiencies

Note: Technicians will assist with these tasks when directed by the CER team manager.

(3) Additional CER Specialists duties are:

• Respond to reconsiderations within five business days of the submission date

• Mentor CER Technicians and assist with analysis assignments and coding inquiries

(4) CER reviewers must use the following when evaluating and analyzing quality:

• EQRS and NQRS reports

• Enterprise Telephone Data (ETD) Reports

• IRM Procedural Updates (IPU) and SERP Alerts

(5) CER reviewers must complete training for the SPRG(s) they review. Some training topics include:

• Embedded Quality review training available through Integrated Talent Management (ITM)

• Quality review command codes

• Integrated Data Retrieval System (IDRS) training modules and related systems

• Coding Consistency cases

• Technical tax law and Interactive Tax Law Assistant (ITLA)

• Correspondence guidelines

• Taxpayer Advocate Service guidelines and criteria

• Oral statement guidelines

• Procedural guidelines

• Communication and Probing skills

• Timeliness guidelines

(6) Quality initiatives, job aids, and training documents for local use only must be approved by the EQ Program Manager. National initiatives, job aids, and training must be approved by headquarters.

(7) Evaluative review and coding questions/problems requiring headquarter assistance must be elevated to the Quality Assurance Specialist (QAS), Process Improvement (PI) Specialist, EQ Program Manager or designee. The QAS, PI Specialist, EQ Program Manager or designee may elevate the issue for additional research/clarification via the Problems/Questions Feedback Form available on EQ-Web. Examples of topics that may be elevated include:

• MAJA definitions or examples

• Quality Gram guidance or examples

• Coding examples not addressed in Quality Grams or by the PI Specialists/QAS requiring assistance from PICA CER Headquarter Analysts

(1) Quality Reviewers will document unauthorized disclosures in the feedback summary section of the DCI using the FLASH procedure (IRM 21.10.1.8.7, EQRS/NQRS Remarks Section) and advise the Operation that they must report the incident per IRM 21.1.3.2.2, Authorized and Unauthorized Disclosures, and IRM 10.5.4.3.3, Inadvertent Unauthorized Disclosures and Losses or Thefts of IT Assets, BYOD Assets and Hardcopy Records/Documents once the Operation accepts the defect or the rebuttal period expires.

(1) The following contains information for the filtering and distribution of IRM Procedural Updates (IPUs) and Alerts posted on SERP. Process Improvement (PI) Specialists and Quality Assurance Specialists (QAS) will designate a minimum of two employees (within each Accounts Management (AM) site); one primary and one backup, to filter and distribute IRM procedures, IPUs and Alerts. Designated employees must:

1. Possess strong technical skills in the applicable programs

2. Have excellent communication skills

3. Can identify information of importance to the site programs and distribute in a timely manner

(2) Distribution will be conducted as follows:

1. Identifying information of high importance requiring immediate distribution (daily)

2. Identifying information that is relevant to site specific programs (weekly)

3. Consolidating information to be shared with the managers (daily/weekly)

(3) Managers will determine what needs to be included in their weekly meetings or can assign to employees as directed read time.

(4) When identifying information for distribution, the designated employee will:

1. Identify the audience or area impacted

2. Use SERP, Tax Forms, and Tax Publication (including IRS websites) to identify information for the employees

3. Distribute the important and relevant information daily/weekly (as described above)

4. For IPU/Alert clarification, the designated employee (Campus/Remote) should contact the PI Specialist

(5) If the PI Specialist needs assistance with IPU/Alert clarification, PI Specialist will contact their local Planning and Analysis (P&A) Analyst responsible for the related program/procedure for assistance.

(6) If after consulting with each other, the PI Specialist and the P&A analyst cannot clarify the procedures, contact the IRM Author of the IPU or the Alert originator by utilizing the SERP Feedback process. PI Specialist will follow guidance under paragraphs (7), (8), and (9) below.

(7) For clarification of an IPU, PI Specialist will submit a SERP Feedback following these steps:

1. Select the topic "IRMs" from the Topics button on the SERP Feedback screen

2. In the "Title" field, indicate "IPU xxUxxxx," using the number of the IPU

3. Refer to the SERP Submitter Guide for information on entering your Feedback issue

4. Follow the guidelines in IRM 21.1.2.2.2, IRM 21 - Change Requests/Servicewide Electronic Research Program (SERP) Feedback.

(8) For clarification of an Alert, PI Specialist will submit a SERP Feedback following these steps:

1. Select the topic "IRM Supplements" from the Topics button on the SERP Feedback screen

2. In the "Title" field, indicate "Alert xxAxxxx," using the number of the Alert

3. Refer to the SERP Submitter Guide for information on entering your feedback issue

(9) For both IPU and Alert clarification, PI Specialist will share a copy of the SERP e-mail confirmation with the PICA Analyst with the assigned SPRG via e-mail. Include with the e-mail:

1. Subject line title: "Request for Clarification"- "IPU xxUxxxx" or "Alert xxAxxxx" (if IPU or Alert)

2. Brief description of the issue (short statement explaining what is unclear and the impact)

3. Supporting documentation if appropriate

4. Copy of SERP feedback control

(10) PI Specialist must keep the Designated employee (who raised the issue) informed that the request for clarification has been raised. PI Specialist will ensure clarification that is received is shared with designated employee for appropriate dissemination locally.

(11) When requesting corrections and/or changes to existing/current IRM procedures, follow the guidelines in IRM 21.1.2.2.2, IRM 21 - Change Requests/Servicewide Electronic Research Program (SERP) Feedback.

(12) Each AM site is responsible for the distribution of SERP information accordingly:

1. Post the Directed Read Time information weekly in a prominent location (e.g., post to local site Quality website). Send a copy via e-mail to Managers and Leads

   Note: PI Specialist should encourage managers to timely deliver the information, especially if the information is critical to quality.

2. Weekly, the designated employee(s) will prepare a consolidated listing for AM employees of all required read items

3. Directed Read Time information will include:
   - The timeframe covered by the information
   - All Alerts and/or Quality Tips should be issued with links to an e-version, if available
   - All information as relevant to the site’s specific programs, any clarification obtained for IRM procedures/IPUs/Alerts and links to the original source
   - Information grouped by specific program or product line based on the audience, when possible. The grouped information must include "ALL," "Balance Due," "Refund Inquiry," "PPS Only," "Phone Only," etc. Directed Read Time information for each week must be archived on the site’s local website for future reference.

Note: Employees are responsible for using official read time each week to review Directed Read Time information.

(13) The Process Improvement Specialist (or designee) will be responsible for the filtering of News Releases made available to the public:

1. Identify special News Releases (i.e., IRS Newswire) of high importance or hot topic that will alert Customer Service Representatives (CSRs) of IRS issues potentially impacting their programs (e.g., Tax Law, Accounts Phone, PPS, Adjustments). These releases will be kept to a minimum to eliminate duplicating each New Release posted on IRS.gov.

2. Identified News Release topics (with the numbers) are to be forwarded to the PICA Team - Accounts Phones SPRG Point of Contact (POC). The PICA POC will send the News Release to SERP for posting on the AM user Portals (IMF/BMF).

3. This process gives the Customer Service Representatives (CSRs) special awareness of certain News Releases distributed to the public. All News Release are posted on IRS.gov.

4. Approved topics will be posted (linked) on the AM Portal “News Release” page.

   Note: The purpose of identifying special News Release topics to post on the AM Portal is to provide additional information for CSR awareness and Training purposes. The IRM, ITLA, and other official IRS publications are to be used to provide information and answers to taxpayer questions.

(1) When creating the Site Level Business Plans, the QAM will:

1. Review past Site Level Business Plans and the current year’s Balanced Measures recommendations to understand the program’s goals.

2. Analyze statistical data, such as NQRS, and EQRS data to identify inconsistencies.

3. Meet with management to discuss QA objectives.

4. Establish QA objectives and set priorities using criteria provided by Headquarters and functional management. Local objectives can be added, if desired.

5. Develop a schedule for implementing each objective of the plan and assign responsibilities.

6. Work with management to prepare a draft of the plan for review.

7. Consolidate suggested changes into a final draft for the Operations Chief’s concurrence, which certifies that the plan is operable.

8. Communicate the applicable portions of the plan to all functional employees, ensuring they understand the plan’s objectives and their roles in the QA process.

(2) After implementation of the Site Level Business Plan, the QAM will monitor plan accomplishments and continue to analyze site data to determine if the operation is meeting objectives. The QAM is expected to recommend needed changes to the plan such as modification or discontinuance of certain objectives.

(1) Following is a list of resources available to the QAM when creating the quality portion of the Site Level Business Plan. This list is in no order and is not meant to be all inclusive. It is designed to suggest the wide variety of data available for consideration.

• Local work plans

• NQRS data including local reports

• EQRS data including local reports

• Timeliness data

• Prior year's Operating Guidelines, including plans for QR and managerial involvement

• Headquarters and functional business plans and reviews

• Field Assistance data

• Accounts Management/Compliance Campus reports on functional activities

• Alert information previously provided to employees

• Staff feedback

• Statisticians feedback or reports

• Taxpayer Advocate Service staff

• Treasury Inspector General for Tax Administration reports

• Results of Improvement Projects

• Focus testing reports

• General Accounting Office reports

• Existing system telephone reports/data

• Customer satisfaction survey results.

(1) The following information is used to determine a sample size:

1. How often quality estimates are necessary (e.g., quarterly, weekly, bimonthly, monthly).

2. The level of precision (e.g., 3 percent, 5 percent).

3. The level of confidence (e.g., 90 percent, 95 percent, 99 percent). (A confidence interval of 90 percent with a precision margin of 5 percent is most often used).

4. A hypothetical "best guess" of the quality rate expected. This guess should be conservative (See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI)) and can be obtained by using historical quality rates. If no historical quality rate exists, 50% should be used.

(2) Unless otherwise specified in IRM 21.10.1, Embedded Quality (EQ) Program for Accounts Management, Campus Collection, Campus Examination, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS), and Electronic Products and Services Support, all cases with a closing count must be made available for the National Quality Review System process. CQRS conducts a National review of the product, not of the employees handling the cases and it may include cases reviewed by managers, on-the-job instructors (OJIs), and cases that are subject to 100 percent review. CQRS focuses on measuring/assessing the taxpayer's experience and their reviews are not used for performance evaluation of employees. There are no exceptions for employees on OJI, new hires, not certified, or reviewed by CER. CQRS will not remove a call or case from the sample because the employee was not certified, a new hire, on OJI, or reviewed by CER or a manager. If the site wants to dispute the assessment or the coding of a call or case, please see IRM 21.10.1.9, Quality Review Rebuttal Process. Cases that are not included in the Operation’s closing count should not be included in the National Quality Review System (NQRS) process.

(3) Sampling assumptions must be determined. Unclear or inappropriate assumptions could lead to a sample that is not random, resulting in estimates that are biased, unrepresentative of the population, or inconsistent. This could call the statistical validity of the estimate into question.

(4) For specified paper product lines, sites must provide sampling assumptions to the Headquarters Process Improvement Customer Accuracy (PICA) analyst responsible for the product. PICA will provide due dates for the sampling assumptions. The site must estimate the total volumes closed for the SPRG for each period. PICA will provide Statistics of Income (SOI) with the sampling assumptions. The SOI staff will calculate the sample size for SPRGs where the samples are SOI designed. IRM 21.10.1.2 (3), The Quality Review Process, to determine if a SPRG has an SOI designed sample.

(5) The following steps are used to apply a skip interval to the population to select a sample. If used correctly, a skip interval will ensure a sample is spread appropriately across the population with estimates that are unbiased.

1. The skip interval is equal to the population, divided by the sample size.

2. Calculate and use a random start number to select the first case (see IRM 21.10.1.3.2.1(6), Selecting the Quality Sample for Sample Plans Designed by SOI). The random start is a randomly selected number that is between 1 and the skip interval. There are a couple of methods for selecting a random number. One such method would be to use Excel's RANDBETWEEN function to generate a random number.

3. The second sampled case is selected by adding the skip interval to the first case selected in step (b). Use this same method to select the rest of the sample (see IRM 21.10.1.3.2.1(7), Selecting the Quality Sample for Sample Plans Designed by SOI).

4. If all the cases are not available at the same time, then sample cases as the work arrives.

5. If all the cases are available at the same time and are stored electronically, then software should be used to sort the list of cases either randomly or by a variable related to the quality measure (e.g., time, case ID) prior to applying the skip interval.

(6) A random start is needed to apply the skip interval. For specified paper product lines where the SPRG has an SOI designed sample, the random start, the skip interval, and the number of sampled cases is determined and provided by SOI as part of the sample plan.

(7) Use the following procedures to select a sample using a skip interval for specified paper product lines:

1. On the first day of the sampling period, use the random start number to identify and select the first sampled case.

2. Use the skip interval to select the subsequent documents for review. In other words, select the ‘nth’ case after the random start case, and continue selecting every ‘nth’ case thereafter.

3. If the population of cases spans more than one day, then the skip interval must continue between days. Begin each new day’s count with the number of cases remaining following the last document selected from the previous day.

Example: Assume a skip interval of "8" and that there were 5 cases remaining after applying the skip interval over Monday’s entire population. Then, continuing the skip interval sequence of 8 into the next day, the case count would begin at 6 on Tuesday. Therefore, the first case selected on Tuesday would be case number 3, the second would be case number 11, the third would be case number 19, etc.

4. A random start number is used only once per sampling plan (at the start of sample selection for that plan). If a skip interval changes in the middle of a sampling timeframe, a new random start should be calculated and used.

5. If the required sample size has been met, continue applying the skip interval through the last case in the population. This will ensure that all work has an equal chance of being selected.

(1) If the site experiences much higher or lower volumes than predicted, the site may change its skip interval within the quarter.

1. For SPRGs that are weighted monthly, the new skip interval may be implemented ONLY at the beginning of a sampling month, NEVER in the middle of a sampling month and only under the guidance of an SOI statistician if the SPRG has an SOI designed sample. IRM 21.10.1.2 (3), The Quality Review Process, to determine if the sample is SOI designed. Because NQRS generates weighted reports monthly, skip intervals must remain constant within any given month.

2. Never simply grab extra cases, drop selected cases, seek out cases of special interest, or use different methods to select cases in the same sample. Each of these situations could lead to a sample that provides biased results.

3. Contact the Headquarters quality analyst for assistance in determining the new skip interval.

(1) In sampling, every sampled case represents a certain number of cases in the population. The exact number of cases a sampled case represents will depend on both the sample size and the actual size of the population from which it was selected. When a quality estimate is a combination of two or more separate samples (e.g., a fiscal year report for a single SPRG for a single site), it is necessary to account for the fact that each sampled case included in the overall estimate may not represent the same number of cases in the overall population. Weighting is used to ensure that every sampled case has the appropriate amount of influence on the overall cumulative estimate.

(2) NQRS provides both weighted and unweighted estimates of quality.

(3) Unweighted estimates that combine more than one site, period, or SPRG are not considered statistically valid. Such estimates should only be used internally. Their statistical limitations should be taken into consideration when basing business decisions on them.

(1) The following steps are used to determine a sample size:

1. How often quality estimates are necessary (e.g., quarterly, daily, weekly, monthly).

2. The level of precision (e.g., 3 percent, 5 percent).

3. The level of confidence (e.g., 90 percent, 95 percent, 99 percent). (A confidence interval of 90 percent with a precision margin of 5 percent is used.).

4. A hypothetical "best guess" of the quality rate expected. This guess should be conservative (See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI)) and can be obtained by reviewing historical quality rates for similar product lines or SPRGs.

   Note: If the resulting sample size is too large for the allocated resources, the Headquarters Product Line Analyst (PLA) may consider reducing the quality estimates. Contact a Headquarters Product Line Analyst (PLA) for assistance with this.

(2) Unless otherwise specified in IRM 21.10.1, Embedded Quality (EQ) Program for Accounts Management, Campus Collection, Campus Examination, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS), and Electronic Products and Services Support, all cases with a closing count must be made available for the National Quality Review System (NQRS) process. This includes cases reviewed by managers, On-the-Job Instructors (OJIs), and cases that are subject to 100 percent review. Cases that are not included in the Operation’s closing count for example certain Compliance Identify Theft cases transferred to Accounts Management, re-routes etc. should not be included in the National Quality Review System (NQRS) process.

(3) Unclear or inappropriate volume counts could lead to a sample that is not random, resulting in estimates that are biased, unrepresentative of the population, or inconsistent. This could call the statistical validity of the estimate into question.

(4) Campus Collection and Campus Examination uses a random selection method that manually spreads the sample among the population as follows:

1. Establish the period of the estimate and the sample size.

2. Compute historical accuracy for each SPRG.

3. Using the sample size and the number of business days in the sampling period, determine the average daily sample size.

4. Depending on the flow of the work for this review, the Product Line Analysts (PLA) will manually spread the sample appropriately among the business days in the quarter. PAS will maintain a record of the sample size and case selection.

5. A calculator with a randomize function will be used to determine the case(s) to be pulled for the day.

(1) In sampling, every sampled case represents a certain number of cases in the population. The exact number of cases a sampled case represents will depend on both the sample size and the actual size of the population from which it was selected. When a quality estimate is a combination of two or more separate samples (e.g., a fiscal year report for a single SPRG for a single site), it is necessary to account for the fact that each sampled case included in the overall estimate may not represent the same number of cases in the overall population. Weighting is used to ensure that every sampled case has the appropriate amount of influence on the overall cumulative estimate.

(2) NQRS provides both weighted and unweighted estimates of quality.

(3) Unweighted estimates that combine more than one site, period, or SPRG are not considered statistically valid. Such estimates should only be used internally. Their statistical limitations should be taken into consideration when basing business decisions on them.

(1) Accounts Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) EIN calls include any questions received on the Telephone EIN application(s) such as.

1. Any call relating to a taxpayer’s request for an Employer Identification Number (EIN)

2. Any call regarding procedural issues (how to complete Form SS-4, Application for Employer Identification Number, where to fax/mail Form SS-4 etc.)

(1) General Account Calls include any questions received on the Telephone Accounts Phones, Advanced Accounts or Procedural applications for General Account Calls. This does not include calls received on the designated Spanish or International Account applications. General Accounts calls include:

1. Any call relating to a taxpayer's account Individual Master File (IMF) or Business Master File (BMF)

2. Any call regarding entity information, the processing of a tax return, corrections to errors found during processing, or corrections resulting from adjustments or audit assessments

3. Any call regarding procedural issues (where to file a return, when and where to make payments, etc.)

4. Any call relating to EIN assignment

(1) SOI develops sampling for Accounts Phones calls monitored at CQRS. Samples from CQRS are valid at the site level bimonthly.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) International Calls include any questions received on the designated telephone applications for International such as:

1. Any international (foreign, non-resident, etc.) call relating to a taxpayer's account (IMF or BMF)

2. Any international call regarding entity information, the processing of a tax return, corrections to errors found during processing or corrections resulting from adjustments or audit assessments

3. Any international call regarding procedural issues (where to file a return, when and where to make payments, etc.)

(1) SOI Staff develops a combined sample plan for International Tax Law and Accounts Phone calls monitored at CQRS. Sample size from CQRS is valid at the site level on a quarterly basis.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) National Taxpayer Advocate (NTA) calls include any calls relating to a taxpayer's account (IMF or BMF) received on the designated NTA Telephone applications.

(1) Managers may use CR and the associated screen shots to evaluate the contact.

(2) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(3) Managerial reviews are not subject to a sampling plan.

(1) PPS Calls include any call from a tax practitioner relating to their client's/taxpayer's account (IMF or BMF) or any other questions received on the Telephone applications for PPS.

(1) SOI develops sampling plans for PPS calls monitored at CQRS. Samples from CQRS are valid at the site and national level on a quarterly basis.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) Spanish Calls include any questions received on the Telephone applications for Spanish Calls including:

• Any call relating to a tax law question from the taxpayer

• Any call relating to a taxpayer's account (IMF or BMF)

• Any call regarding entity information, the processing of a tax return, corrections to errors found during processing, or corrections resulting from adjustments or audit assessments

• Any call regarding procedural issues (where to file a return, when and where to make payments, etc.)

(1) SOI Staff develops a combined sampling plan for Spanish Tax Law and Spanish Accounts calls monitored at CQRS. Samples from CQRS are valid (which includes Spanish Accounts and Spanish Tax Law combined) at the site level monthly. For all other sites combined, the statistical validity is monthly.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) Accounts Management (AM) Identity Theft Calls include any call from an individual to report that their SSN or ITIN has been misused to obtain goods or services, to report other complaints of identity theft, and/or request protection of their tax account information or any questions received on the Telephone ID Theft Phone application(s). For additional information, please see IRM 25.23.12, IMF Identity Theft Toll-Free Guidance.

(1) SOI develops sampling for ID Theft Phone calls monitored at CQRS. Samples from CQRS are valid at the site level quarterly.

(2) Ultra (Contact Recording) and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) The AM Routing Product line consists of one Specialized Product Review Group (SPRG), Default Screener Calls.

(2) AM Routing phone calls are reviewed to measure and improve the accuracy, timeliness and professionalism of transfers to the appropriate area.

(1) Default Screener phone calls are defined as any calls received on the designated Telephone Default Screener applications.

(1) SOI Staff develops sampling plans for AM Default Screener phone calls monitored at CQRS. Sample sizes from CQRS are valid at the national level monthly.

(2) CR and the associated screen shots may be used to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to sampling plans.

(1) AM Routing Default Screener phones will be measured for Timeliness, Professionalism, Regulatory/Statutory Accuracy, and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) The Forms Order Phones Product Line calls will be reviewed to measure and improve the quality of our responses to taxpayers’ request for forms and answers to procedural questions.

(2) The Forms Order Phones Product line consists of one Specialized Product Review Group. The SPRG is National Distribution Center (NDC).

(1) Forms Order Phones is defined as any call received by the vendor.

(1) NDC phone calls are defined as any calls received by the vendor.

(1) SOI Staff develops a sampling plan for NDC phone calls monitored at CQRS. The sample is a daily sample and is valid at the national level monthly.

(1) NDC Form Order calls will be measured for Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) The Tax Law Phones Product Line calls will be reviewed to measure and improve the quality of responses to taxpayers’ tax law and technical questions.

(2) The Tax Law Phone Product line consists of three Specialized Product Review Groups. The three SPRG's are General Tax Law Calls, International Tax Law calls, and Spanish Tax Law calls.

(1) Tax Law Phones is defined as any General, International or Spanish calls received on the Telephone Tax Law applications.

(1) For Accounts Management, all Tax Law Phone SPRGs will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy, and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) General Tax law phone calls are defined as any on-line calls received on the designated Telephone Tax Law applications. This does not include calls received on the designated Spanish or International Tax law applications.

(1) SOI staff develops sampling plans for General Tax Law Phones calls monitored at CQRS. The sample is a daily sample and is valid at the site and national levels on a quarterly basis.

(2) CR and the associated screen shots may be used by CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(1) International phone calls are defined as any on-line calls received on the designated Telephone International Tax Law applications.

(1) SOI Staff develops a combined sample plan for International Tax Law and Accounts Phone calls monitored at CQRS. Sample size from CQRS is valid at the site level on a quarterly basis.

(2) CR and the associated screen shots may be used by CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(1) Spanish Tax Law Phone calls are defined as any call received on the designated Spanish Telephone Tax Law Applications.

(1) SOI staff develops a combined sampling plan for Spanish Tax Law and Spanish Accounts Phone calls monitored at CQRS. Sample sizes from CQRS are valid at the site and national level monthly.

(2) CR and the associated screen shots may be used by CQRS to evaluate the contact.

(3) A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) The TE/GE Phones Product Line will be reviewed to measure and improve the quality of responses to taxpayer's/customer's technical and account related questions.

(2) The TE/GE Phones Product line consists of one Specialized Product Review Group (SPRG).

(1) TE/GE Phones calls are defined as any tax law, procedural or account calls received on the designated Telephone TE/GE applications.

(1) SOI Staff develops sampling plans for TE/GE phone calls monitored at CQRS. Sample sizes from CQRS are valid at the site level on a bimonthly basis.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) The site may want to perform local reviews to aid in the quality improvement of the product line. Local sampling guidelines have been included. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(1) TE/GE Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory Accuracy and Procedural Accuracy. These are the measures that are available and reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) Collection and Examination Campus Quality Phone SPRGs will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) The following are the Collection Campus Quality Phone Product Lines:

• ACS Phones

• Automated Substitute for Return (ASFR) Phones

• Centralized Case Processing (CCP Collection Phones and Centralized Lien CLO Phones)

• Centralized Insolvency Operation (CIO) Phones

• Collection Phones

• Private Debt Collection Phones

(1) The following are the Examination Campus Quality Phone Product Lines:

• Automated Underreporter (AUR)

• CS Specialized Phone Services

• Exam Phones

• Innocent Spouse Phones

(1) e-help Phone calls are defined as any calls received on the designated Telephone e-help applications.

(1) SOI Staff develops sampling plans for e-help phone calls monitored at CQRS. Sample sizes from CQRS are valid at the site level monthly.

(2) CR and the associated screen shots may be used by managers or CQRS to evaluate the contact.

(3) The site may want to perform local reviews to aid in the quality improvement of the product line. Local sampling guidelines have been included; See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(4) Managerial reviews are not subject to a sampling plan.

(1) e-help Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory Accuracy and Procedural Accuracy. These are the measures that are available and reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) TSO Phone calls are defined as any calls received on the designated Telephone TSO Customer Service Section applications.

(1) Managers may use CR and the associated screen shots to evaluate the contact.

(2) The site may want to perform local reviews to aid in the quality improvement of the product line. Local sampling guidelines have been included; IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), and IRM 21.10.1.3.2.1, Selecting the Quality Sample for Sample Plans Designed by SOI.

(3) Managerial reviews are not subject to a sampling plan.

(4) SOI Staff develops sampling plans for TSO phone calls monitored at CQRS. Sample sizes from CQRS are valid at the site level monthly.

(1) TSO Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory Accuracy and Procedural Accuracy. These are the measures that are available and reported under the Balanced Measurement System. IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) Review the finished product for Tax Account Activity work (which includes some types of employee generated correspondence) from IRS Received Date to the Closing Date for the inquiry or internally generated case. Do not include in the review any subsequent or corrective actions taken after the Closing Date.

(2) Review each sample case and enter identification data on the applicable section of the NQRS/EQRS DCI. All sections of the NQRS/EQRS DCI apply for Accounts Paper except the Exam section.

(1) Any cases worked under Tax Account Activity are subject to Accounts Paper (Adjustments, AM Identity Theft Paper and Return Preparer Misconduct) review. This includes electronic Forms e-4442, and written Forms 4442, Inquiry Referral, and Taxpayer Correspondence.

(2) A case is considered Tax Account Activity work if it requires the employee to either research an account or perform adjustment action on an account. All Tax Account Activity work completed within any campus site is to be included in one SPRG sample plan for the campus. Separate sample plans for each remote site would be cost prohibitive.

(3) Examples of types of Tax Account Activity work included in the Accounts Paper reviews:

• Internal Notices/CPs (see IRM 3.13.62, Media and Transport Control, for a complete list of internally generated CP Notices)

• TCB/NOL–Tentative Carrybacks

• Previous Adjustments

• Amended Returns (Form 1040X, 1120X, 94X)

• CP59

• Loose Schedules

• Duplicate filed returns

• EFTPS Issues

• FTDs

• Penalty Issues

• Wage Economic Impact Payment (WEIP) Mailbox e-mail

(4) Examples of Tax Account work excluded for Accounts Paper reviews:

• Criminal Investigation

• Collection

• Examination

• Phone Calls

• E-mail

• ELF

• Undeliverable Mail

• Unidentified Remittances in Receipt and Control

• Automated Extensions (i.e., Form 4868, Form 2688, etc.)

• Bank Letters–Outgoing

• Receipt and Control Payments

• Large Corp Cases (including PPRG cases)

• Cases routed to another area

• CAWR

• Installment Billing

• Form 1133 – Not a closed case

• Form 3858 – Not a closed case

• ASFR

• Form 8871 (OSC Only)

• Returns being statute cleared and routed to another area (710–01028),

• Entity Unpostables

• Form 8655, Reporting Agent Authorization for Magnetic Tape/Electronic Filers

• Reworking an Unpostable

• CII case that was closed as MISC and linked to another open CII case

• CII case that was closed under the wrong program (For example: Submission Processing working a 1040 MEF case, closed the case under the Accounts Paper Adjustments SPRG)

• Split image unrelated case that are erroneously processed as one single CII case. Refer to IRM 21.5.1.5.1(16), CII General Guidelines. Splitting the image cannot be evaluated/coded on DCI as part of case selection for reviews. The unrelated case did not appropriately fall into the sample pull.

• Case selected for sample review is unreviewable (requiring replacement case) i.e., case linked to an open case on day of closure.

Note: The above Included and Excluded lists are not all inclusive. Contact Headquarters for a decision on whether to review any Tax Accounts Activity work not found on either list.

• Cases closed and reopened on a later day due to corrective action are not reviewable per IRM 21.10.1.5.1.1, Roles and Responsibilities of the Accounts Paper Analyst, because cases reopened after the original closing date are unreviewable per 21.10.1.5.1.1(2), Roles and Responsibilities of the Accounts Paper Analyst.

(5) The following will help you determine the received date:

• Determine whether your case is a Taxpayer Initiated Inquiry (TPI) or non-TPI. When you are unable to determine whether your case is a TPI case, or other type of case, treat the case as a TPI.

• When there is no IRS Received Date or handwritten received date on the document, determine the received date following the guidance in IRM 21.5.1.4.2.4, Received Date - Determination.

  Note: For information concerning the Statute Reprocessable Return, see IRM 21.5.2.4.23.4, Statute Imminent Documents.

• On computer-generated cases, use the transcript or notice date minus 14 days to determine the IRS received date. If no computer-generated date is present, use the IRS received date on the source document.

  Exception: A CP 98/198 case resulting from a TPI should be controlled using the IRS received date of the source document.

  Exception: A CP 193 should be controlled using the transcript received date.

• When fax receipts are date and time stamped by the fax machine, no further action is required to evidence receipt. Otherwise, fax receipts received before 2:00 p.m. local time, are considered received on the current date. Documents received after 2:00 p.m. local time are considered received the next business day.

(1) Headquarters Quality Product Line Analysts (PLA) provide sample sizes for written referral review at the sites. Statistics of Income (SOI) develops the sample plan when requested by Headquarters. This plan allows for statistically reliable estimates quarterly at the site level and monthly at the national level.

(2) For non-CII Adjustments work, the sites should use the skip for AMSS Support Services. The Adjustments Non-CII SPRG is not measured and is input into NQRS only.

(1) Accounts Paper will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy, and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) AM Specialized Services is reviewed to evaluate and improve the quality of account actions and correspondence in response to written taxpayer inquiries and internal requests.

(1) AM Clerical – Campus Support encompasses "live" review of the Campus Support mail operation and evaluates each process for efficiency and timeliness to ensure that the objectives are met. The objectives are as follows: To open and extract mail; expedite all remittances to Deposit; minimize time from receipt of mail through the sorting function to routing; identify and control all payments for shipping; and expediting.

(1) e-help Support Paper is defined as any paper or e-mail case closed by the e-help Operation. This includes all correspondence (paper and electronic) received directly from a customer or a telephone call reduced to writing via an Interaction/Incident closed by the e-help Operation.

(1) Campus Quality Paper SPRGs will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) Collection Campus Quality Paper Product Lines include:

• ACS Written

• Automated Substitute for Return (ASFR) Paper

• Centralized Case Processing (CCP) Collection Paper and CLO Lien Paper

• CIO Paper

• Offer in Compromise (OIC) Paper

• Collection Paper

• Independent Administrative Review

(1) Examination Campus Quality Paper Product Lines include:

• Automated Underreport (AUR) Paper

• CCP Exam Paper

• CS Specialized Paper Services

• Exam Paper

• Innocent Spouse Paper

(1) TE/GE Correspondence Analysts will use the TE/GE Correspondence skip interval to select sample cases each day from the previous day's closures.

(2) TE/GE Correspondence Analysts will complete an unbiased, consistent and accurate review of closed TE/GE Correspondence Unit paper cases, covering all technical, procedural, and professional actions taken.

(3) DQ any letter that will result in erroneous information being sent to a Taxpayer. See IRM 2.4.5, Command Codes QRADD, QRADDO, QRNCH, QRNCHG, RVIEW, QRACN, and QRIND for the Quality Review System.

(4) Review the entire case file, including any clerical processes to identify the actions taken. Ensure there is a corresponding annotation on the history sheet that includes all the required information (date, time, explanation of actions taken) and that the work papers thoroughly support the conclusion of the case.

(5) For each case reviewed, complete an NQRS/EQRS DCI, and all appropriate attribute fields, indicating whether the quality standards were met, while providing as much information as possible, including all IRM references for coded defects. This review should identify defects as outlined in IRS Regulations and IRM guidelines and procedures, and provide a basis for error analysis and error trends.

(6) Enter the word FLASH in Remarks in the Finish section of the DCI to designate an error case that requires immediate corrective action by the site. Return the case to the examiner's Manager for correction. See IRM 21.10.1.8.7, EQRS/NQRS Remarks Section, for more information.

(7) Input the review data daily using NQRS.

(8) Refer to the Embedded Quality SharePoint site (https://irsgov.sharepoint.com/sites/eqrsc-support) weekly to glean updated information on the use of attributes in the TE/GE Correspondence Unit, obtain the latest Quality Gram, review the Issue Log, etc.

(9) In a monthly report, which is to be shared with the TE/GE Correspondence Unit, provide suggestions for improvement by:

• Identifying most frequently made defects and the top five Customer Accuracy Drivers by SPRG

• Analyzing root causes of defects

• Reviewing guidelines or sampling procedures

• Reviewing methods used to capture needed information

(1) TE/GE Correspondence is defined as any paper case closed by the TE/GE Correspondence Unit. This includes all paper correspondence received directly from a Taxpayer or a telephone call reduced to writing via Form 4442 closed in the TE/GE Correspondence Unit.

(2) TE/GE Correspondence is based on two primary sources:

• Direct Taxpayer Correspondence

• Form 4442 referrals

(1) Samples from the site are valid at the site and national level on a quarterly basis.

(2) The analyst must sample and review the complete closed TE/GE Correspondence case, (after all input actions are completed), including any clerical process performed on the product.

(3) National Review will consist of a random sample of cases, based on the skip interval.

(4) SOI does not develop a sampling plan for TEGE Correspondence.

(5) Local reviews are not performed for the national measure of the TE/GE Correspondence product line and therefore are not included in the sampling plan. However, if time and staffing permits, the site should also perform local reviews to aid in the quality improvement of the product line. See IRM 21.10.1.3.2, Quality Review Sampling Guidelines for Sample Plans Designed by Statistics Of Income (SOI), for more information on local reviews.

(1) TE/GE Correspondence will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy, and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.8.3, EQRS/NQRS Measures, for more information on the measures.

(1) TSO Paper is defined as any paper or e-mail case closed by CSS in TSO. This includes all correspondence (paper and electronic) received directly from a customer or a telephone call reduced to writing via an Interaction or Incident.

(1) The HQA is responsible for the overall planning, administration, and evaluation of the quality-related sections of the Site Level Business Plans. The HQA will identify problems and work with management to solve them.

(2) The EQRS review process, when combined with NQRS data and other functional data, will assist the HQA in evaluating the overall quality of operations and making recommendations for improvement.

(3) The AQA will serve as the Quality (QA) Analyst for the Area, ensuring that designated quality resources are used to focus on quality improvement efforts.

(4) The AQA/HQA will maintain the integrity and quality of the QR system by monitoring and reviewing NQRS results.

(5) By using trend analysis, the AQA/HQA will determine the causes of recurring quality defects. The AQA/HQA will assist management in initiating programs to help employees improve the quality of FA service to taxpayers. It is important that lines of communication remain open among the AQA/HQA, management and CQRS to identify problem areas, take appropriate corrective actions, and re-evaluate quality to ensure that corrective actions result in improved quality.

(6) The AQA/HQA will act as the liaison between the CQRS staff and FA management. The AQA will advocate for the Area when submitting rebuttals and is responsible for communicating quality information to all managers in the Area.

(7) The AQA/HQA will identify training needs and recommend to the Training Coordinator and/or management, the type of training needed (such as on-the-job training, online courses, informal workshops, or classroom instruction), and assist in the development of additional training exercises and workshops to meet those needs.

(8) The AQA/HQA and the CQRS staff will be responsible for the protection of NQRS DCIs and any supporting documentation from legacy systems. All documents and information (including taxpayer information) seen, heard, or handled must remain secure and confidential.

(9) The AQA will serve as the Embedded Quality liaison with managers and will work with managers to accurately assess performance and code EQ attributes, provide continuous and substantive performance feedback and recognize and reward quality at all levels (e.g., employee-of-the-month, Quality Honor Roll, etc.), using the National Agreement for guidance.

(10) The AQA/HQA will be responsible for working with the Training Coordinator and/or management to ensure that EQ training is made available to all who need it and advise if instruction in training materials are consistent with service quality standards and objectives.

(1) Attribute 715 Customer Accuracy will be rated on each contact unless the event being reviewed is internal work that would never have impact on the customer's case, account, or issue. In that situation consider Attribute 718, Met Internal Requirements. See Exhibit 21.10.1-6 and Exhibit 21.10.1-7 for assistance with coding attributes 715 and 718.

(2) Attribute 715 Customer Accuracy should never be coded "N" by itself. There should be another Regulatory and/or Procedural attribute that caused the Customer Accuracy error. These other attributes are referred to as Customer Accuracy Drivers. It is these "driver attributes" that are used to make improvements. Prior to the completion of the DCI, a window will appear showing all the attributes that were coded "N" during the review. To complete the DCI, a decision must be made to determine the core defect causing attribute 715 to be coded "N". You can choose multiple drivers of Customer Accuracy; however, you must choose a minimum of one attribute to define the reason for coding attribute 715 as "N." When selecting the driving attribute, consider the following:

• What specific attribute(s) had a direct effect on the decision to code attribute 715 as "N"?

• Is this the primary attribute that, if corrected, would improve the Customer Accuracy defect?

(1) Defect Reason Codes were developed for attributes in the EQ system that either have a high error rate or that cover multiple activities. These defect reason codes help clarify errors coded by identifying the root cause of the error. When an attribute with defect reason codes is coded "N," a drop-down menu will appear listing defect reason codes for this attribute. You will be required to select at least one code that best describes the root cause of the error. You can select as many codes that fit the specific situation. If none of the codes fit the root cause of the error, select [Other]. Refer to the MAJA or Quality Job aid on the EQ SharePoint site, https://irsgov.sharepoint.com/sites/eqrsc-support, for your SPRG for more information on defect reason codes.

(1) There are three input fields to record remarks in EQRS.

(2) EQRS Accounts Management and RIVO only - Enter the words FLASH or Corrective Action Required at the beginning of the Feedback Summary remarks if a defect requires corrective action by the site. If the defect doesn’t impact the taxpayer, such as an internal procedure, do not use FLASH or Corrective Action Required notations.

• Use FLASH when a defect requires immediate action to avoid further impact to the customer.

• Use Corrective Action Required when a defect requires action to avoid further impact to the customer but does not require immediate action.

(1) There are three input fields or "boxes" to record remarks in NQRS. These are Customer Issue, Employee Actions/Response and Feedback Summary. Each box has a maximum number of characters including spaces that may be input.

(2) The CQRS, PAS analysts and Campus Examination PAS analysts will enter the words FLASH or Corrective Action Required at the beginning of the Feedback Summary remarks field to designate an error case that requires immediate corrective action by the site. If the defect doesn’t impact the taxpayer, such as an internal procedure, do not use FLASH or Corrective Action Required notations.

• Use FLASH when a defect requires immediate action (e.g., imminent refunds, incorrect adjustment) to avoid further impact to the customer.

• Use Corrective Action Required when a defect requires action (e.g., input a STAUP, send a return to be processed, send a response letter, etc.) to avoid further impact to the customer but does not require immediate action.

(1) Reports on the EQRS side of the database are in three categories, Employee Reports, Organization Reports, and Ad hoc Reports. Ad hoc Reports are discussed in IRM 21.10.1.8.13, EQRS/NQRS Ad Hoc Reports. The following reports are available in EQRS:

• View/List Single Case DCI Report (Employee)

• Individual Review Feedback Report (Employee)

• Cumulative Review Feedback Report (Employee)

• Organization Cumulative Report (Organization)

• Top Ten Defects/Successes Report (Organization)

• Customer Accuracy Driver Report (Organization)

• Days to Close Report (Organization)

• Time in Inventory Report (Organization)

• DCI Deletes/Changes Report

• EQRS/NQRS Comparison Report

(2) The View/List Single Case DCI Report includes all the DCI header fields including the Taxpayer TIN. This report is for reference only and should never be filed in the Employee Performance File (EPF).

(3) In EQRS, the quality attributes are mapped to employee critical job elements (CJEs) and the performance aspects within those CJEs. The CJEs for the various SPRGs can be found on the EQ Website. A key concept of Embedded Quality is to link employee performance to the business measures. By coding employee’s quality using mapped attributes the employee's performance is tied directly to the business measures.

(4) There are two types of employee evaluative reports generated from this mapping that are available through EQRS - the Employee Individual Review Single Case Feedback report (for a single case review), and the Cumulative Review Feedback report (case reviews over a period of time).

• "The Employee Individual Review Single Case Feedback Report" shows the quality of that case by bucket. It also groups each CJE by aspects and shows coded attributes within each aspect. Each coded attribute shows whether it was marked "Y" or "N." All three Remarks fields from the DCI appear on the report - Customer Issue, Employee Actions/Response, and Feedback Summary. There is a signature block for the Manager, Reviewer (if not the manager), and Employee. If the report is generated based on non-evaluative reviews, the bucket information will be shown but the CJE mapping information will not.

• "The Cumulative Review Feedback Report" shows the quality of all reviewed cases over a designated period by bucket. It also groups each CJE by aspect and shows coded attributes within each aspect. Each coded attribute shows the number of times it was rated Y, the number of times it was rated N, and an overall Yes percentage. The Feedback Summary field from each of the reviewed cases during the period will be shown on the report, with a hyperlink back to the Individual Review Feedback report for that case. There is a signature block for the Manager and Employee.

(5) There are several organizational level reports that are unweighted. The organizational reports are as follows:

• The Organizational Cumulative Report can be generated for any organizational segment down to a team level depending on the user’s permission level. The report displays the overall bucket information for the timeframe selected as well as the percentage of times each attribute was met. There are several filters that can be selected to refine the report output to more specific information. Filters are review date range, Site, BOD, SPRG, bucket, case type, group code, job series, OFP, reasons for contact, resolution type, review category and reviewer type.

• The Top Ten Defects/Successes Report displays the top ten attribute defects or successes for the organizational levels and types of work over a period. It identifies both positive and negative trending and can sort accuracy information by either number of defects, total opportunities, or percentage errors. Reports can be generated down to the team level depending on the permission level of the user. There are several filters that can be selected to refine the report output to more specific information. Filters are review date range, site, BOD, SPRG, bucket, case type, group code, job series, OFP, reasons for contact, resolution type, review category and reviewer type.

• The Customer Accuracy Driver Report is designed to assist the user in determining the attributes that are driving Customer Accuracy errors. The report will display a list of the top attributes that are causing the Customer Accuracy errors. The attributes selected as drivers make up the pool of Customer Accuracy Drivers that are displayed in the output of the Customer Accuracy Driver Report in descending order of frequency.

• The Days to Close Report (Unweighted) displays the Average Days in Department or the Average Days in Inventory. Days in Department is calculated from the Operation/Territory Received Date to the Closing Date and the Days in Inventory is calculated from the IRS Received Date to the Closing Date. A roll up by BOD or Enterprise can be selected, or one or more sites can be selected. The report is available by Product Line or SPRG. The average days to close is displayed as a total and with a break down by group code.

• The Time in Inventory Report displays various ranges of dates: 1 to 14, 15 to 30, 31 to 44, and 45+ and then shows the number of cases closed in these ranges and the percentage of the total cases closed in the sample. The report shows data for the cases within the review date range input. The report is available by Days in Department or Days in Inventory. The Days in Department is calculated as Operation/Territory Received Date to Closing Date. The Days in Inventory is calculated as IRS Received Date to Closing Date. This report is used for paper product lines and can only be run for one Product Line or SPRG at a time.

• The DCI Deletes/Changes Report displays the cases that were deleted or edited that meet the filter criteria input. The report displays the DCI number, the date the record was deleted or changed, the name of the user who deleted or changed the record and the reason for delete or change selected. There are several filters that can be used to limit the output including site, date range, delete/change reason code, delete/change date, delete/changed by, organization, reviewer and SPRG.

• The EQRS/NQRS Comparison Report displays attribute coding comparisons between EQRS and NQRS for the selected SPRG. There are various filters that allow the user to customize reports. The primary filters that can be used are SPRG, site and date range. There are also some additional filters available.

Note: For Campus Collection and Campus Examination, the applicable NQ Site Types must be selected if different from EQRS SPRGs. See IRM 21.10.1.8.5, EQRS/NQRS DCI Header, for a list of site types.

(1) Various weighted and unweighted reports are available in NQRS for site and Headquarters personnel with NQRS access. Most Specialized Product Review Groups (SPRGs) and some Product Lines have monthly weighted reports available, and all have monthly unweighted reports. The weighted reports are available in NQRS in the following metrics:

• Customer Accuracy

• Timeliness

• Professionalism

• Days to Close

(2) The unweighted reports available on NQRS are:

• View/List DCI

• Quality Attribute Accuracy

• Quality Rate

• Accuracy by Header Field

• Days to Close

• Time in Inventory

• Top Defects/Successes by Site

• Customer Accuracy Driver

• Systemic Error Driver Report

• DCI Deletes/Changes

• Transfer Report

• Volumes Report

Note: Unweighted accuracy is defined as the total number of "Yes" coding divided by the total number of opportunities (Yes and No Coding).

(3) Weighted reports are those where weighting specifications have been developed by the Statistics of Income (SOI) office. For those SPRGs where SOI designs the sampling plan a minimum sample is provided for each site and each Product Line or SPRG that will result in a statistically valid sample of the work performed for a specific period. The standard weighting formula uses the sample and volume of work performed in each Product Line or SPRG at each site as well as the number of total opportunities and the number of correct opportunities in each metric to weight the review results. This allows the results for a given Product Line or SPRG to be proportionally represented with a precision margin. A precision margin is a measure, usually in the form of a percentage that describes how close the values of a sample estimate are to the true population values. Volume is input for each SPRG every month beginning on the first day and ending on the tenth day after the end of the reporting period.

(4) Volume is input for paper SPRGs every month beginning on the first day and ending on the tenth day after the end of the reporting period.

• All the telephone SPRGs in Accounts Management and many of them in Campus Collection and Campus Examination have daily volume transferred each week automatically from ETD. These figures can be updated manually if necessary. The telephone SPRGs that do not have automatic transfer of daily volume must have these volumes input manually which can be done throughout the month.

• The Field Assistance SPRGs input volumes on a weekly basis.

• For the Campus Collection and Campus Examination Paper SPRGs, PAS will input volumes daily using the count derived from the universe of cases available for the sample selection.

(5) Weighted Reports are automatically run each Saturday or Sunday for the preceding month but will not generate for a specific Site/SPRG combination until the volume has been input for that Site/SPRG prior to the reports running. Volume input between Monday and Friday will be reflected on the report that runs on Saturday or Sunday. The Product Line Small Business Self Employed (SBSE), Taxpayer Services (TS), Directorships and/or and Enterprise roll-ups will not be accurate until the volume is input for all Sites where the SPRG is worked. All reports are weighted at the SPRG and/or Product Line level. Accounts Phones – Spanish and Tax Law Phones – Spanish are weighted together as a product line as are Accounts Phones – International and Tax Law Phones – International.

(6) NQRS reports are all based on pre-defined reporting periods for each SPRG. SPRGs use either review date, closing date or sampled date and calendar month or cycle month to define the month for reporting purposes. See Exhibit 21.10.1-11 for a chart of NQRS Reporting Periods. Exhibit 21.10.1-9 and Exhibit 21.10.1-10 for Cycle and Calendar Cut-off Dates. A cycle month begins on the last Sunday of the preceding month and ends on the last Saturday of the current month. For reports where a date range is input, the cases retrieved are either between two review dates or two closed dates depending on the SPRG. Where a report period is selected from a drop-down list, the cases retrieved are based on closed date or review date as well as calendar month or cycle month depending on the reporting period for the SPRG. If you are trying to match the information on one of the canned reports such as the Quality Attribute Accuracy Report or Quality Rate Report with an Ad hoc Report, it is important to understand the predefined reporting periods of the specific SPRG you are analyzing.

(7) All the NQRS reports are available in HTML and Excel. Following are reports available in NQRS:

• The Weighted Customer Accuracy Report provides customer accuracy information weighted by volume for several product lines and/or SPRGs. Some work types are weighted at the product line level while others are only weighted at the SPRG level or some other combination depending on the business rules in place. The report can be generated at the SBSE, TS, Directorship, enterprise, or site level. If you select a site or sites, you will see the information specific for the site(s) selected as well as the weighted information for the appropriate SBSE, TS and the enterprise. A weighted report can be generated, excluding Systemic Defects by selecting Y in the Excluding Systemic Error field.

• The Weighted Timeliness Report provides timeliness information weighted by volume for several product lines and/or SPRGs. Some work types are weighted at the product line level while others are only weighted at the SPRG level or some other combination depending on the business rules in place. The report can be generated at the SBSE, TS, directorship, enterprise, or site level. If you select a site or sites, you will see the information specific for the site(s) selected as well as the weighted information for the appropriate SBSE, TS and the enterprise.

• The Weighted Professionalism Report provides professionalism information weighted by volume for several product lines and/or SPRGs. Some work types are weighted at the product line level while others are only weighted at the SPRG level or some other combination depending on the business rules in place. The report can be generated at the SBSE, TS, directorship, enterprise, or site level. If you select a site or sites, you will see the information specific for the site(s) selected as well as the weighted information for the appropriate SBSE, TS and the enterprise.

• The Days to Close Weighted Report provides weighted information on the average number of days that a case was in the inventory of a specific Operation/Territory before it was closed or the average number of days it was in the IRS inventory before it was closed. The Days in Department is calculated as Operation/Territory Received Date to Closing Date. The Days in inventory is calculated as IRS Received Date to Closing Date. This report is only available for specific paper SPRGs.

• The View/List DCI Report is the report generated to view the coding of a single DCI. The report shows the bucket scores on a case as well as the Attribute and Header field coding breakdown and the remarks input. This report is reviewed daily for each review at the site level.

• The Quality Attribute Accuracy Report displays information by attribute for a specific product line or SPRG at a specific site, or for a SBSE, TS or Enterprise. The report displays each attribute including the number of opportunities, the number of times it was coded as correct, the number of times it was coded as incorrect and the accuracy percentage for the attribute. The output can be displayed in Accuracy Measurement or DCI Tab (Opening, Research, etc.) order. The report parameter screen includes filters for Review Type (National or Local), Site, Product line or SPRG, Group Code, OFP, AIMS Status, and Application Number. The reports available are shown in a drop-down list that includes months as well as weeks. If a month is selected, a Planning Period and Fiscal Year to date roll up will also be included.

• The Quality Rate Report displays the quality rate for all five quality buckets for the timeframe specified in the query. The report will display data for the site(s) selected, the appropriate BOD(s) and the Enterprise. The report displays the product line/SPRG selected, sample size, opportunities, number correct, number incorrect and the accuracy percentage for each quality bucket. The output displays the total for the sites selected individually and, as well as the Enterprise total for the Product Line or SPRG selected. The individual site data is also shown broken out by group code. The Enterprise or SBSE, TS data can be shown by group code if selected as a roll up.

• The Accuracy by Header Field Report is designed to display the quality rate for all five quality buckets for the period selected with the ability to display the information by specific header fields. A total is shown as well as a breakdown by the header field selected. For example, you can ask for an Enterprise report for Accounts Phones General and ask that the output be broken out by application number. You will see the bucket data for each application as well as a total for the period selected. The report can be run for a single site, several sites at once, or with a SBSE, TS or Enterprise roll up. The report periods are available by week ending or by month. If requested by month, a planning period and fiscal year roll up will also be provided.

• The Days to Close Report (Unweighted) displays the Average Days in Department or the Average Days in Inventory. Days in Department is calculated from the Operation/Territory Received Date to the Closing Date, and the Days in Inventory is calculated from the IRS Received Date to the Closing Date. A roll up by SBSE, TS or Enterprise can be selected, or one or more sites can be selected. The report is available by Product Line or SPRG for most paper Product Lines/SPRGs. The average days to close is displayed as a total and with a breakdown by group code.

• The Time in Inventory Report displays various ranges of dates, 1 to 14, 15 to 30, 31 to 44 and 45+ and then shows the number of cases closed in these ranges and the percentage of the total cases closed in the sample. The report shows data for the month selected and the planning period and fiscal year. The report is available by Days in Department or Days in Inventory. The Days in Department is calculated as Operation/Territory Received Date to Closing Date. The Days in Inventory is calculated as IRS Received Date to Closing Date. This report is used for paper product lines and can be run for one SPRG at a time.

• Top Defects/Successes by Site displays the Top 5/10 or All Defects/Successes for the parameters input. The user can request that the report be displayed by the greatest number of Defects/Successes, Total Opportunities or by Percentage. The user can request all measurement types together (default) or request the output for one specific measurement type (Timeliness, Professionalism, Customer, Regulatory, or Procedural Accuracy). The user can also request the output be broken down by a specific header field or no breakdown (default). The header field options are ACS Priority Code, AIMs Status, Application Number, Case Type, CP Number, Disp/Cert Code, Group Code, Local Use, MFT Code, National Use, OFP or Resolution Type.

• The Customer Accuracy Driver Report is designed to assist the user in determining the attributes that are driving Customer Accuracy errors. The report will display a list of the top attributes that are causing the Customer Accuracy errors. Each time the Customer Accuracy Attribute is coded "N," the user is prompted to select the attributes that were coded N, that the user determined to be the main cause of the Customer Accuracy error. There is no limit to the number of attributes that can be selected as Customer Accuracy Drivers. These attributes that make up the pool of Customer Accuracy Drivers are displayed on the Customer Accuracy Driver Report in descending order of frequency.

• The systemic Error Driver Report (available only for Compliance SPRGs) displays a list of attributes that were identified as systemic errors when the Systemic Error header field is indicated with a “Yes.” The user is prompted to select the attribute(s) that were coded N because of a systemic error. There is no limit to the number of attributes that can be selected as Systemic Error Drivers. These attributes that make up the pool of Systemic Error Drivers are displayed on the Systemic Error Driver report in descending order of frequency.

• The DCI Deletes/Changes Report displays the cases that were deleted or edited that meet the filter criteria input. The report displays the DCI number, Product Line, SPRG, Reviewer number, Changed/Deleted date, Changed/Deleted by, Change/Delete reason code, and Changed/Deleted Notes. There are several filters that can be used to limit the output including review type (Local or National) site, date range, delete/change reason code, delete/change date, delete/changed by, organization, reviewer, and SPRG.

• The Transfer Report displays the number of transfers from a specific phone application for the SPRG selected. It will also display the percent of the sample transferred for each application and the number and percent of the transfers that were appropriate or not. The report is for analyzing phone transfers.

• The Volumes Report displays the Volume Date, Volume entered, and the sample reviewed for that date of the specific SPRG/Site selected. The report can be generated using a predefined period or a date range. This report is for analyzing sample vs. volumes by date.

(1) There are four types of ad hoc reports available in EQRS/NQRS.

• List Reports display a list of the reviews in the database that meet the criteria outlined in the filters selected. The information displayed on the report will depend on the items the user selects to be displayed. There is no limit on the number of fields displayed on a List report.

• Review Count Reports display a count of the reviews in the database that meet the criteria outlined in the filters selected. The information displayed is limited to six fields and will be displayed in the order selected.

• Bucket Count Reports are the same as count reports except that they display the bucket scores for each of the fields selected.

• Reason Code Reports are used to display the number of times and the percentage of time each reason code was selected for a specific attribute. Multiple attributes can be selected at a time. The attributes are selected from the attribute filter section.

(2) The difference between the report types is in the output display. For all report types, the queries are the same. Each time you run a new report you must first define the filters and then define what you want displayed in your output. When you define the filters of your query you are asking the database to display a list or count of the reviews that meet the criteria you define. When you define your output, you are defining the items that will be displayed on the report. The order of the display is especially important on the count report and can change the whole meaning of the output.

(3) It is important to understand the distinction between defining filters and defining output. When you define filters, you are telling the database what data you want and consequently what data you do not want to see. By selecting a range of review dates for example, you are telling the system you only want to see reviews with review dates between the dates selected. This is true for all filters selected. If you select 715=N as a filter, you are telling the system you only want to see data on cases where 715=N. The more filters you use the less data your report will retrieve. If you use 715=N and 508=N, you are telling the system you only want to see data where both 715 and 508 equal N. If you wanted to see how attribute 715 was coded when attribute 508 was coded as N, you would use 508 = N as a filter and display 715.

(4) The display is only telling the system what you want to see from the database on the cases that meet the filter criteria you selected and in what order you want to see it. If you use 715=N as a filter there is no sense in displaying 715 because you know it will always = N on the cases displayed on the report. If you do not use 715 as a filter at all you would want to display 715 to see if it was Y, N, or I on the cases on the report. If you use the SPRG filter and only select one SPRG then you would not need to display the SPRG because you already know what it is. If you do not use the SPRG filter or multi-selected SPRGs, then you would want to display the SPRGs so you can see which one related to the DCI. An exception to this might be on a bucket count report. You may want to display the SPRG and then the Group Code for example. This would show you the bucket counts for the data requested for the SPRG and then show the bucket counts broken out by Group Code.

(5) You will look at the display items you select differently depending on which type of report you are building. Each review (DCI) in the database contains a lot of information. Some of that information may not be relevant to the output you are looking for. Although you can display all the information related to a review with an Ad hoc List Report, it is usually not practical or desirable to do so.

(6) Once you build a report that meets your needs it can be saved and run periodically whenever you need the data. Often, you will want to redefine one or more of the filters each time you run the report (i.e., Review Date Range, Organizational Segment, etc.).

(7) There are six basic steps in the Ad-hoc report building process.

1. Determine what information you need.

2. Determine the report type you want: List, Review Counts or Bucket Counts.

3. Determine which filters you need to use to get the information you need.

4. Determine what you want to be displayed on the report and in what order.

5. Run the report and review the information.

6. Save the report as either private or public.

(8) Ad hoc Reports are generated from the Reports Menu in NQRS or EQRS. The following list provides more detail to the report building process.

1. Select the Ad hoc Reports menu by clicking on Ad hoc Reports.

2. Select the Type of Report you wish to generate, then, select Define Filters. Remember, you are asking the database to display a list or count of the reviews that meet the criteria you define.

3. You can filter the data with any field in the database making these reports true Ad hocs that allow you a great deal of flexibility in pulling information from the database. The filters may seem daunting at first, but you will learn that you only use 3 to 5 filters on most reports. There are four categories of filters, or fields, within the database. Depending on what type of entry, each field is categorized as one of the following:

Filter Type	Filter Explanation
Range Item	Fields where your filters are defined as a range. Most of these fields are date ranges such as Review Date and IRS Received Date. When you select one of these dates you will be asked to input a To: and From: date. By doing this you are specifying that you want to retrieve, as a List or Count, the reviews that fall within the date range specified.
Multiple/List Select Items	Items that must be selected from a predetermined list of values within the database. This leaves less room for error from a data entry standpoint and makes these fields reliable when pulling data based on the specified values.
Entry Items	Items that are input by the users when they are entering a DCI or defining filters for an ad hoc report. Only the number of characters and sometimes the type of characters limits these fields, which makes them a little less desirable to be used as filters from a data integrity standpoint.
Attributes	Include all the attributes in the database and are not limited to the attribute within the Smart DCI for a product line. When you select an attribute as a filter you must also select "Y," "N," or "N/A." In other words, do you want a list or count of each time the attribute was coded Y, N, or N/A. You can also multi-select these values. For example, if you wanted to see cases where attribute 508 was coded you would select 508 as a filter and then select Y and N. This would filter out any cases where attribute 508 was not coded.

4. After you have selected the type of report and defined your filters it is time to determine what items you want displayed on the report. Click "Items Displayed on Report." To select an item to be displayed, double click on the item from the available items list. After you double click on the item it will move to the Items Selected list.

5. To display attributes on the report you have several options. First, you can select attributes one at a time from the available attribute list. You can also select all the attributes by selecting [All Attribute Groups] from the attribute group list although this is not usually a desirable option because you will see many attributes that are never coded and therefore will always be inapplicable. You can also select the attributes to display by attribute group (Opening/TPI, Research/Referral, etc.). The last option is to display the "smart" or "total" attributes for a specific SPRG; and/or the CA Drivers (A715) can be selected for multiple SPRGs. The smart and total selection will automatically populate the output with all attributes that are in the DCI for a specific SPRG and can only be used if one SPRG is selected as a filter. You also have this option with header fields.

6. Items selected to be displayed on the report will display in the order selected. If you want to change the order of the display, de-select and re-select items until you have them in the order you want them displayed. You can also alter the display order with the sequence numbers to the right of the display list.

   Note: There is a limit of six items that can be displayed on a Count or Bucket Count Report. There is no limit to the number of fields that can be displayed on a List Report, but you may want to limit the number of fields displayed if you plan to print the report.

(9) Once you have selected the Report Type, Defined the Filters and Defined the Items to be displayed on the Report, all you have left to do is to Name, Save, and Run the report. You can also go back later and retrieve or delete a previously saved report.

1. The name you give the report should be as descriptive as possible so you will know what information is included in the report immediately. Once an ad hoc report is built that meets your needs, you want to run the report periodically, just changing one or more of the filter items such as the date range.

2. Ad hoc reports can be saved as "Private" or "Public." Only the user who created the report can view a private report. Any user can view a public report if they have report access. In EQRS there are some limits to viewing Public reports due to permission issues.

3. To run an ad-hoc report, select Show Report from the Ad-Hoc Report Menu. While you are building the report, you can run it as many times as you like to ensure that it includes the information you need. The report is not saved until you select Save Report as discussed above. If you select Excel as the output format you may receive a dialog box asking if you want to open or save the file. You would select the option to open the report. When running reports in Excel it is important to close the report before you run it a second time. If you don’t then you can have problems with Excel because you may have two files open with the same name which can be problematic. If you are running a report several times while fine tuning it to give you the information you are after, you should just run the report in HTML until it is final to avoid this problem.

4. To retrieve a report, select Retrieve a Stored Report from the Ad hoc Report Menu. The Ad hoc Reports-Stored Reports Menu will be returned. You will see a list of the ad hoc reports you have created. To see public reports created by others select View Public Reports.

5. To delete a report, there is a button on the ad Hoc report screen where the user's stored reports are displayed. To delete a stored report, click on the report name and then click on the [Delete] button. Another method is to click once on the report from the Ad hoc Reports from the Stored Reports menu and then hold down the CTRL key and select Up Arrow on your keyboard. You can only delete reports that you have created.

(1) The Help Page includes four links to choose from:

• Master Attribute Job Aid (MAJA)

• EQ Web Site

• SERP Web Site

• IRM 21.10.1, Embedded Quality (EQ) Program for Accounts Management, Campus Collection, Campus Examination, Field Assistance, Tax Exempt/Government Entities, Return Integrity and Compliance Services (RICS), and Electronic Products and Services Support

(2) Master Attribute Job Aid (MAJA): The MAJA is an electronic Job Aid that contains information specific to the SPRG selected from the MAJA Home Page. Once you select an SPRG the MAJA will display the Header Items and Attributes that are in the Smart and Total DCIs for the SPRG. For each field it also displays the Operational definition and the Job Aid definition that is found on the Job Aids for each SPRG. The attribute mapping is displayed as Smart, Total, Discretionary, or Required. The bucket mapping is also displayed and for those attributes with reason codes there is a link that will list the reason codes mapped to the selected SPRG and attribute. If you access the MAJA from the Help screen on EQRS you will also have a link that shows the CJE mapping for each job series on EQRS.

1. If there are coding examples listed for an attribute for the SPRG selected there will be a link below the attribute that can be used to view the examples.

2. The Blank DCI for each SPRG is located on the upper right of the screen on the MAJA once an SPRG is selected. If you access the MAJA from EQRS you will select the appropriate Job Series prior to displaying the blank DCI and the CJE mapping will be reflected on the blank DCI. You can also indicate if you want to see only the Smart DCI attributes or if you want to display the Total DCI. On the NQRS side there is no need or option to select the Job Series prior to displaying the Smart or Total Blank DCI.

3. There is a Print Job Aid button available on the MAJA that will display a job aid that can be printed.

4. You can also see coding examples on the MAJA for the SPRG being reviewed by selecting the Print Example button.

5. The MAJA for EQRS and NQRS can also be accessed from the EQ SharePoint site. The URL for the EQ SharePoint site is: https://irsgov.sharepoint.com/sites/eqrsc-support.

(3) The Embedded Quality SharePoint site is the main communication tool for the EQ program. It is located at https://irsgov.sharepoint.com/sites/eqrsc-support. Managers and reviewers should be familiar with how to navigate the EQ SharePoint site and they should check it frequently.

(1) First level rebuttals must be received by the designated Headquarter Analyst via E-mail no later than ten business days from the review date. Rebuttals received after the ten-day timeframe will not be considered.

(2) The first level rebuttal must include:

• DCI number

• Attribute(s) being rebutted

• IRM or Policy Statement cited

• Explanation of rebuttal

• No personally identifiable information (PII) or sensitive but unclassified (SBU) data

(3) If the submitter still disagrees with the original defect, the Department Manager can initiate a second level rebuttal.

(1) Within five business days of the first level response the Department Manager may submit a second level rebuttal to Quality via e-mail. Rebuttals received after the five-day timeframe will not be considered.

(2) The second level rebuttal must include:

• DCI number

• Attribute(s) being rebutted

• IRM or Policy Statement cited

• Explanation of rebuttal

• No personally identifiable information (PII) or sensitive but unclassified (SBU) data

(3) If the submitter disagrees, the Department Manager may initiate a meeting with the Quality Manager and the Headquarter Analyst(s) to discuss the review. The meeting must take place within five business days of the second level response.

(1) It is essential that all sites and functions follow the same guidelines for coding quality service. To ensure National (NQRS) and Embedded (EQRS) Quality reviews are based on appropriate and current procedures, the following procedures are effective immediately.

(2) The procedures for preparing and implementing an IRM deviation are as follows:

• Prior to implementation any guidance that deviates from the IRM or that establish new practices for temporary procedures or pilot projects must:

1. Be in written format (e-mail, memo etc.)

   Note: Verbal instructions and meeting minutes notes do not constitute a formal deviation and will not be considered during the quality review process.

2. State the reason for the deviation

3. What caused the situation to occur

4. What is being done to correct it

5. The beginning and end date of the deviation (not to exceed one year)

   Note: Deviations are only good for the time specified but never exceed one year. Deviations are not retroactive and become effective after all applicable parties have been notified in writing. During the deviation period the work will be reviewed based on the procedures outlined in the deviation. 

6. Distribution of the deviation memo must include RIVPM SOA, BPL, HQ P&A, HQ Program Management, HQ Quality, RIVO Director and RIVO P&A

(3) Quality defects related to IRM or procedural changes will be charged ten- business days after notification of the deviation and/or SERP posting date of the IRM update/change. During the ten-business day grace period, national analysts will code "Y" if either the former or new procedure is followed. Informational feedback from local and national reviews will be shared with functional areas prior to the tenth day.

(4) Suspension or early termination of a deviation must be in written format and shared timely with RIVPM SOA, HQ P&A, HQ Program Management, HQ Quality, RIVO Director and RIVO P&A staff.

(5) Active/expired deviations used during the phone and/or paper product lines and SPRG reviews will be posted on the Quality for RIVO SharePoint under the IRM Deviations tab.

(1) To allow time for CQRS to perform a consistency review of all CQRS DCIs, Sites should not submit a rebuttal until at least two business days after the DCI is input.

(2) Rebuttals must be submitted on Form 14448, Quality Review Rebuttal. See the CQRS Website (https://irsgov.sharepoint.com/sites/cqrscentralizedqualityreviewsystem/SitePages/CQRS-Web.aspx) for the CQRS Rebuttal Contact and appropriate e-mail address. The site should ensure that all applicable fields on the form are correct and complete. Follow the instructions on the CQRS web site to complete the form. The site is required to submit an unedited copy of the DCI in PDF format with the disputed attributes along with the Form 14448.

(3) Rebuttals of CQRS defects must be received by CQRS no later than ten business days from the week ending date the review was input to NQRS.

(4) Before submission of a rebuttal, management should review NQRS to determine if the DCI had been edited during the review period.

(5) Before submitting a rebuttal, when possible, management must review the contact via CR to ensure the facts presented in the rebuttal are correct.

(6) The Area Quality Analyst (AQA) will contact CQRS via E-Mail regarding the disputed defect. The HQ Quality Analyst (HQA) will provide the E-Mail address.

(7) The following information must be included with the rebuttal:

• DCI Number

• Site information

• Reason for disagreement with the defect

(8) The CQRS analyst, senior analyst, and/or section chief will consider the rebuttal and any back-up research provided. They may consult with the Headquarters Product Line Analyst at any time during the rebuttal process.

(9) CQRS personnel will respond to rebuttals within ten business days from receipt of rebuttal.

(10) Second-level Rebuttals must be submitted within ten business days from CQRS's first-level response. Allow ten business days for CQRS to respond to second-level rebuttals.

(11) Third-level Rebuttals must be submitted within five business days from CQRS's second-level response.

(12) The HQA will review all documentation and will determine whether the defect was appropriately identified and will respond to the AQA. The HQA will consider management and CQRS statements, and will discuss the issues with the AQA, CQRS, and the EQ Core Team Lead as needed. The HQA makes the final determination of whether a defect has been appropriately identified. Management must provide any research material used when reviewing the assessment. If the HQA disagrees with management, they will respond to the site. If the analyst concurs with management, they will forward the case and supporting documentation to CQRS and the AQA. The referral must contain the DCI number of the disputed case and supporting E-mails and documentation from the site and CQRS.

(13) A record copy or original copy of the rebuttal, and all supporting documentation, must be maintained for the current year and prior fiscal year. After that time, the records can be destroyed.