10.8.62. Information System Contingency Plan (ISCP) and Disaster Recovery (DR) Test, Training, and Exercise (TT&E) Process

(1) Organizations must develop and operate a testing program in non-disaster situations so that IRS leadership and personnel have familiarity with contingency plans and procedures and validates the IRS’ contingency capabilities through regular tests, training, and exercises. It can also identify issues or deficiencies for remediation. (IRS-defined)

(2) Exercises and tests offer different ways of ensuring that ISCPs provide viable and actionable procedures to recover or restore IRS systems and applications to their original state in the event of a disruption. (IRS-defined)

(3) Steps to establish a Test, Training and Exercise (TT&E) program should include the following: (NIST 800-84: Chapter 2)

1. Develop TT&E policy.

2. Identify TT&E roles and responsibilities.

3. Establish overall TT&E schedule.

4. Document TT&E methodology for planning and performing TT&E events.
   i. Design the event - topic, scope, roles and responsibilities and objectives.
   ii. Develop the event documentation - may include briefing materials, participant manuals, instructor and facilitator guides, test plans, and evaluation criteria.
   iii. Conduct the event.
   iv. Evaluate and document lessons learned from the event.

(4) The following elements are suggested to be included in a TT&E policy: (NIST 800-84: Section 2.1)

• Purpose

• Effective date

• Objectives

• Applicability and scope

• Authorities and related policies

• Roles and responsibilities of key business units and staff positions

• TT&E requirements

• TT&E review and approval

• Enforcement and compliance

• Points of contact for additional information

• Definition of terms

(5) Refer to National Institute of Standards and Technology (NIST) Special Publication (SP) 800-84, for guidance on establishing an effective ISCP testing program and the various methods and approaches for conducting exercise activities. (IRS-defined)

(6) All tests and exercises must include some kind of determination of the effects on the organization’s operations and provide for a mechanism to update and improve the plan as a result. (IRS-defined)

(7) The depth and rigor of ISCP testing activities increases with the FIPS 199 availability security objective. Refer to the ISCP templates (FIPS 199 LOW, MODERATE, and HIGH systems) in NIST SP 800-34 Contingency Planning Guide for Federal Information Systems, for details for conducting testing activities appropriate to their respective impact level. (IRS-defined)

(8) The depth and rigor of ISCP testing activities increases with the FIPS 199 availability security objective. (IRS-defined)

• For LOW and MODERATE-availability systems, a tabletop and functional exercise must be conducted annually to ensure that a basic level of recovery capability is available for all reportable assets within these categories. The tabletop should follow a scenario that simulates a disruption, include points of contact whose roles appear in the ISCP, be attended by the business and system owners or responsible authority, and be facilitated by ISCPT personnel. The functional exercise must be performed by IRS IT or BOD IT personnel.

• For HIGH-impact systems, Critical Infrastructure Protection assets, or assets that support any of the MEFs, a tabletop exercise and full-scale end-to-end or DR test must be conducted annually to ensure that a full recovery capability is available for the most critical reportable assets. The tabletop should follow a scenario that simulates a disruption, include points of contact whose roles appear in the ISCP, be attended by the business and system owners or responsible authority, and be facilitated by ISCPT personnel. The full-scale test should include a system restoration at the designated alternate location. The test must also include a full recovery and reconstitution of the system to a known state.

(1) The ISCP must provide procedures and capabilities for recovering a system or application in the event of a system disruption. The plan must address the resources, roles, responsibilities, and procedures for restoration of information systems and recovery of business applications and processes after a disruption. (IRS-defined)

(1) The keystroke procedures located in the ISCP are an information system-focused part of the plan designed to restore operability of the target system or application, at an alternate site after an emergency. (IRS-defined)

(2) The purpose of the keystroke recovery procedures is to provide documented, detailed step-by-step procedures to facilitate recovery of capabilities at an alternate site; the scope is information system-focused and limited to major disruptions. (IRS-defined)

(1) Tabletop exercises are discussion-based exercises only and do not involve deploying or recovering systems, equipment, or other resources. (NIST 800-84: Chapter 4)

(2) An ISCP tabletop exercise is a simulation of an emergency designed to validate the viability of one or more aspects of an ISCP. (NIST 800-84: Chapter 2)

(3) Personnel with roles and responsibilities in a particular ISCP should be invited to attend the tabletop exercise to validate the content of the plan during a discussion of their roles and responses to emergency situations, execution of responses in a simulated operational environment, or other means of validating responses that do not include using the actual operational environment. (NIST 800-84: Chapter 2)

(4) ISCP tabletop exercises are scenario-driven (such as a power failure in one of the organization’s computing centers or a fire causing certain systems to be damaged) with additional situations often being presented during the course of an exercise. (NIST 800-84: Chapter 2)

(5) ISCP tabletop exercises help to identify gaps and inconsistencies within ISCPs and procedures, as well as cases where personnel need additional training or when training needs to be changed. Deficiencies identified during tabletop exercises are documented in the ISCP Testing Checklist as part of the exercise process. (NIST 800-84: Chapter 2)

(6) Information or procedures needing validation during the ISCP tabletop exercise include but are not limited to roles and responsibilities, hardware and software inventories, keystroke recovery procedures, data back-up procedures, and interdependencies that are documented in the plan. (NIST 800-84: Section 4.2.3)

(7) An ISCP tabletop exercise schedule must be created annually by the ISCPT staff. BOD POC’s must be notified and their approval obtained by the ISCPT staff when a tabletop exercise is being scheduled. (NIST 800-84: Section 4.1)

(8) IRS must conduct ISCP tabletop exercises for applicable applications or systems at least annually. Applicable applications or systems include those that are classified as Tier 1, 2 or 3. (IRS-defined)

(9) The topics should be determined. Discussion topics may include the roles and responsibilities of personnel with regard to disaster recovery and incident response. Discussion topics may also include processes and procedures for disaster recovery and incident response. (NIST 800-84: Section 4.2.1)

(10) The scope should be determined. Senior-level teams and operational-level teams should participate in separate tabletop exercises initially because of their different levels of responsibility and then in a combined exercise to validate coordination between the groups. (NIST 800-84: Section 4.2.2)

(11) Application or system ISCP Leadership and operational-level personnel, including a data collector (as designated by the BOD), should be invited to ISCP tabletop exercises to discuss individual and team roles, responsibilities, and to validate information contained in the plan. (NIST 800-84: Section 4.2.4)

(12) The ISCPT facilitator leads the discussion among the tabletop exercise participants. (NIST 800-84: Section 4.2.5)

(13) Logistics for ISCP tabletop exercises should be coordinated. (NIST 800-84: Section 4.2.6)

(14) The tabletop exercise material should be developed. (NIST 800-84: Section 4.3)

(15) The ISCP Testing Checklist is an IRS internal document designed to assist BODs and support staff in navigating through tabletop exercise events. The BOD designated data collector records information and actions that occur during the exercise using an applicable version of the ISCP Testing Checklist. Refer to Exhibit 10.8.62-1 and 10.8.62-2 at the end of this document. (NIST 800-84: Section 4.2.5; IRS-defined)

(1) Functional exercises allow personnel to validate application/system operational readiness for emergencies by validating backup procedures – specifically with regards to backup retrieval, reading backup data, and validation of offsite storage.(NIST 800-84: Chapter 5)

(2) Functional exercises allow staff to execute their roles and responsibilities as they would in a recovery situation. (NIST 800-84: Chapter 5)

(3) A functional exercise schedule should be created annually by the ISCPT staff and shared with Data Management and BOD IT staff. The following should be identified: (NIST 800-84: Section 5.1)

1. Topics (NIST 800-84: Section 5.2.1)

2. Scope (NIST 800-84: Section 5.2.2)

3. Objectives (NIST 800-84: Section 5.2.3)

4. Participants (NIST 800-84: Section 5.2.4)

5. Functional exercise staff (NIST 800-84: Section 5.2.5)

(4) Data Management staff in IT and BOD IT must conduct functional testing using the following frequency: (IRS-defined)

1. Quarterly for FIPS 199 HIGH systems.

2. Semi-annually for FIPS 199 MODERATE and LOW systems.

(5) Logistics for functional exercises should be coordinated. (NIST 800-84: Section 5.2.6)

(6) The functional exercise material should be developed. (NIST 800-84: Section 5.3)

(7) Data Management staff within IT and BOD IT must provide evidence of conducted functional tests, to include evidence of media retrieval from an offsite storage location and screen shots to validate successful testing has been completed. (IRS-defined)

(8) Data Management staff in IT and BOD IT must record functional test results using Part B of the applicable version of the ISCP Testing Checklist. (IRS-defined)

(9) Evidence gathered from functional tests should be submitted to the Cybersecurity, ISCPT staff within five (5) work days from the test ending date. (IRS-defined)

(10) Evidence packages for all applications/systems receiving a functional test will be uploaded to TFIMS by the Cybersecurity, ISCPT staff before the end of each FISMA cycle. (IRS-defined)

(1) A DR test is the method used to evaluate the organization's readiness and ability to recover an application or system from varying degrees of non-functioning to its original functional state in an alternate operational environment specified in an ISCP. (IRS-defined)

(2) DR Tests are used to measure the effectiveness and suitability of the processes and procedures contained in ISCPs for the systems being tested and to evaluate compliance with a contingency plan. In the event of a disaster or disruption, the goal is to use ISCPs to ensure that documented operational procedures and plans result in successful recovery of business applications and systems. (IRS-defined)

(3) The scope of tests can range from individual system components or systems to comprehensive tests of all systems and components that support an ISCP. (NIST 800-84: Section 6.2.1)

(4) A test is conducted in as close to an operational environment as possible, testing components, or systems used to conduct daily operations. (NIST 800-84: Section 6.3)

(5) A DR test is required annually for CIP assets, assets that support any of the MEFs or assets with a high availability categorization. (IRS-defined)

1. IRS must conduct two Enterprise Computing Center (ECC) tests at least annually – one at ECC-Martinsburg in Kearneysville, West Virginia, and the other at ECC-Memphis in Memphis, Tennessee. Evidence from successful ECC tests will satisfy annual DR testing requirements for several CIP, MEF and/or high availability assets.

2. CIP, MEF and high availability assets not included in the ECC tests must be scheduled for individual DR tests annually. Evidence from successful individual tests will satisfy annual DR testing requirements for CIP, MEF and/or high availability assets not included in the ECC DR tests.

3. Ad hoc DR test requests for applications/systems other than CIP, MEF or high availability assets must be evaluated on a case-by-case basis by EOps, ITCM.

(6) A DR test schedule should be created and managed collaboratively between the Cybersecurity, ISCPT staff and BOD IT staff annually for DR tests needed to satisfy DR testing requirements for CIP, MEF or high availability assets managed by BOD IT personnel. (NIST 800-84: Section 6.1)

(7) Each reportable application or system involved in a DR test must have a contingency plan that includes but is not limited to: (IRS-defined)

1. Environment description;

2. Host system information (if applicable);

3. Interconnecting system dependencies;

4. Alternate processing site information;

5. Hardware/Software inventories;

6. Keystroke recovery procedures (if applicable);

7. Configuration information;

8. Backup information;

9. Escalation and notification procedures; and

10. Key personnel contact list(s)

(8) The DR test event should be designed. The following should be identified:(NIST 800-84: Section 6.2)

1. Objectives (NIST 800-84: Section 6.2.2)

2. Testing tools (NIST 800-84: Section 6.2.3)

3. Participants (NIST 800-84: Section 6.2.4)

4. DR test staff (NIST 800-84: Section 6.2.5)

(9) The DR test material should be developed. (NIST 800-84: Section 6.3)

1. Briefings are developed for senior management, and for the managers of others that might be affected by the test, to provide an understanding of what the test will comprise and why it is important.

2. The test guide outlines the basic steps involved in conducting a test and includes a list of the participants. Procedures for early termination of the test should be included.

3. Test plans list steps that will be performed, required logistical items, expected outcomes, early test termination procedures, and emergency contact numbers.

4. The after action report, or Summary Report, contains an overall synopsis of the DR test, the results of individual tests, and the recommendations for improvement. This report may be provided to senior management.

(10) Each DR test for CIP, MEF or high availability assets managed by EOps must be facilitated by the EOps, ITCM staff. EOps, ITCM personnel are responsible for: (IRS-defined)

1. All aspects of the DR exercises, including staffing, development, conduct, and oversight;

2. Appointing test director(s); and

3. Coordinating logistics. Sample logistic items for DR tests may include but are not limited to the following: (NIST 800-84: Section 6.2.6)
   i. DR test date(s)
   ii. Identification of applications/systems for testing
   iii. Participant identifications
   iv. Meeting invitations
   v. Conference room reservation and set-up
   vi. Setup and configuration of appropriate testing equipment
   vii. Required testing tools
   viii. Backup file strategies
   ix. Dry-run/walk through of the test
   x. Procedures to terminate the test

(11) Each DR test for CIP, MEF or high availability assets managed by EOps must be observed by the Cybersecurity, ISCPT staff. (IRS-defined)

(12) DR tests required for CIP, MEF or high availability assets managed by BOD IT personnel must be conducted by BOD IT staff. (IRS-defined)

(13) Evidence gathered from each DR test must include: (IRS-defined)

1. Screen shot(s) of the Production and Recovery environment including the server names in each environment.

2. Screen shot(s) of production data copied (reloaded) into the Recovery environment.

3. Screen shot(s) from the Recovery environment showing the start and stop times of the server(s) to calculate the actual recovery time(s).

4. Keystrokes and/or failover plan from Toolkit Suite Command Centre (TSCC) or playbook.

5. Testing/validation of the restored data, batch runs, end user testing etc.

6. Documentation of any issues/problems encountered. (NIST 800-84: Chapter 2)

7. Documentation of action taken to resolve issues/problems encountered. (NIST 800-84: Chapter 2)

8. List of participants.

(14) Evidence gathered from DR tests should be submitted to the Cybersecurity, ISCPT staff within fifteen (15) work days from the test ending date. (IRS-defined)

(15) The Cybersecurity, ISCPT staff must produce a test report presentation for the DR tests at ECC-Martinsburg and ECC-Memphis. (IRS-defined)

(16) The ECC test report presentation must include: (IRS-defined)

1. An ISCPT Observation Report of the test for Senior Management and Executives (refer to part (20) below).

2. Evidence gathered during the test, including screen shots.

3. A vulnerabilities matrix listing tickets submitted using the IRWorks/ServiceNow ticketing system for problems identified and actions taken to resolve issues.

(17) Evidence packages for all applications/systems tested during the DR test will be uploaded to the TFIMS by the Cybersecurity, ISCPT staff before the end of each FISMA cycle. (IRS-defined)

(18) The ISCPT Observation Report is a presentation for Senior Management and Executives. This overview is a summary report of an ECC DR test which contains an overall synopsis of the recovery capabilities. The ISCPT Observation Report may include: (IRS-defined)

1. Scenario

2. Test Objectives

3. Summary Test Results

4. Test Scope

5. Commendable Observations

6. Gaps Identified

7. Cybersecurity Recommendations

8. Problem Ticket Information

9. ISCPT Team Contact Information

(19) Refer to Exhibit 10.8.62-5, ECC Disaster Recovery (DR) Test Job Aid. This exhibit provides step-by-step procedures for IRS IT personnel to perform the ECC DR Tests. All ECC Disaster Recovery (DR) Tests will be conducted using the approved procedures in Exhibit 10.8.62-5. (IRS-defined)

(1) An Alternate Site Processing (ASP) test may be conducted in lieu of a Disaster Recovery test. (IRS-defined)

(2) During an ASP test, the application/system that is transitioned will run at the alternate site for a predetermined period of time. (IRS-defined)

(3) An ASP test schedule must be managed by the Enterprise Operations (EOps), IT Continuity Management (ITCM) Branch staff for each ASP test EOps conducts. This schedule may be combined with the DR test schedule. (IRS-defined)

(4) The ASP test schedule must be shared with the Cybersecurity ISCPT staff by the EOps staff annually. (IRS-defined)

(5) Each reportable application or system involved in an ASP test must have a contingency plan that includes, but is not limited, to the following information: (IRS-defined)

1. Environment description;

2. Host system information (if applicable);

3. Interconnecting system dependencies;

4. Alternate processing site (APS) information;

5. Hardware/Software inventories;

6. Keystroke recovery procedures (if applicable);

7. Configuration information;

8. Backup information;

9. Escalation and notification procedures; and

10. Key personnel contact list(s)

(6) Each ASP test for CIP, MEF or high availability assets managed by EOps must be facilitated by the EOps ITCM staff. EOps ITCM personnel are responsible for: (IRS-defined)

1. All aspects of the ASP exercise, including staffing, development, conduct, and oversight;

2. Appointing test director(s); and

3. Coordinating logistics. Sample logistic items for ASP tests may include, but are not limited, to the following:
   i. ASP test date(s)
   ii. Identification of applications/systems for testing
   iii. Participant identifications
   iv. Meeting invitations
   v. Conference room reservation and set-up
   vi. Setup and configuration of appropriate testing equipment
   vii. Required testing tools
   viii. Backup file strategies
   ix. Dry run/walk through of the test
   x. Procedures to terminate the test

(7) Each ASP test for CIP, MEF or high availability assets managed by EOps must be observed by the Cybersecurity ISCPT staff. (IRS-defined)

(8) ASP tests required for CIP, MEF or high availability assets managed by BOD IT personnel must be conducted by BOD IT staff. (IRS-defined)

(9) Evidence gathered from each ASP test must include: (IRS-defined)

1. Screen shot(s) of the Production and Alternate Processing Site (APS) environments including the server names in each environment.

2. Screen shot(s) of a successful replication of data from the Production environment to the APS.

3. Screen shot(s) from the APS environment showing the start and stop times of the server(s) to calculate the actual recovery time(s).

4. Keystrokes and/or failover plan from TSCC or playbook.

5. Testing/validation of the restored data, batch runs, end user testing etc.

6. Documentation of any issues/problems encountered. (NIST 800-84: Chapter 2)

7. Documentation of action taken to resolve issues/problems encountered. (NIST 800-84: Chapter 2)

8. List of participants.

(10) Evidence gathered from ASP tests should be submitted to the Cybersecurity ISCPT staff within 15 work days from the test ending date. (IRS-defined)

(11) Evidence collected for applications/systems tested during an ASP test will be uploaded to the TFIMS by the Cybersecurity ISCPT staff before the end of each FISMA cycle. (IRS-defined)

(12) Refer to Exhibit 10.8.62-3, Ad Hoc Disaster Recovery/ASP Testing SOP. This exhibit provides step-by-step procedures for IRS IT or BOD Information System personnel to perform Disaster Recovery/ASP exercises. All Ad Hoc Disaster Recovery/ASP Testing will be conducted using the approved procedures in Exhibit 10.8.62-2. (IRS-defined)

(1) Training refers to informing personnel of their roles and responsibilities within an information system contingency plan and teaching them skills related to those roles and responsibilities, thereby preparing them for participation in exercises, tests, and actual emergency situations related to the information system contingency plan. (NIST 800-84: Chapter 3)

(2) The scheduling of training sessions will be coordinated closely with the schedules for ISCP tabletop exercises, functional exercises, and DR tests. (NIST 800-84: Chapter 3)

(3) Training sessions will emphasize understanding the ISCP Testing process, to include following documents in preparation for participating in each test or exercise: (IRS-defined)

1. ISCP – Participants will be able to answer questions about the purpose of the plan, system recovery procedures, specific application processes, recovery roles and responsibilities, notification procedures, and all appendices included in the plan.

2. ISCP Testing Checklist – Participants will gain knowledge of the purpose of the Checklist, how to complete it, and the procedures for its use during the scheduled exercises and tests of the ISCP.

3. FISMA Contingency Plan (CP) Controls – Participants will gain knowledge of the Contingency Plan family of security controls (NIST 800-53A, Guide for Assessing the Security Controls in Federal Information Systems and Organizations) and how exercising and testing the contingency plans address the CP controls.

(4) Recovery personnel must be trained on the following plan elements: (IRS-defined)

• Purpose of the plan

• Reporting procedures

• Security requirements

• Activation and Notification, Recovery, and Reconstitution Phases

(1) The ISCP Testing Checklist is a three part form that allows BODs and Support Organizations to document multiple ISCP exercise/test results on one form to create a testing artifact that includes AO authority. (IRS-defined)

1. Part A of the Checklist is the Tabletop Exercise

2. Part B is the Functional Exercise

3. Part C is the Disaster Recovery (DR) Test or Production Operational Recovery which documents DR Testing activities. Refer to Exhibit 10.8.62-1 for a copy of the Checklist. (IRS-defined)

(2) The ISCP Testing Checklist provides a step-by-step process to guide participants through the most pertinent sections of the ISCP. The Checklist provides an area to document changes for each section in the ISCP and changes to procedures that might be needed. The Checklist also provides areas to document the results of functional exercises and DR tests, if applicable. (IRS-defined)

(3) The Checklist standardizes the process for all applications and systems, and documents all testing activities and ISCP changes. The Checklist serves as the validated artifact for ISCP testing, functional exercises, and DR Testing exercises and events. The Checklist and supporting documentation are uploaded to TFIMS after it has been reviewed and signed by the AO or AODR. (IRS-defined)

(4) The Checklist is used to train personnel in their contingency roles and responsibilities with respect to their application or system. (IRS-defined)

(5) Completion of the Checklist documenting performance of the required ISCP testing exercises and/or tests serves as an evidentiary artifact in TFIMS to verify that the following family of controls, if appropriate (Reference NIST 800-53): (IRS-defined)

1. CP-02 Contingency Plan – The ISCP is pulled from TFIMS and distributed to each participant for the tabletop exercise validating that the plan exists.

2. CP-03 Contingency Training – The requirements, roles and responsibilities, and recovery procedures are discussed during the ISCP tabletop exercise.

3. CP-04 Contingency Plan Testing and Exercises – Completion of the ISCP Testing Checklist with test dates and the AO’s signature and signature date serves as evidentiary documentation that the ISCP was exercised and appropriate testing was conducted during the applicable FISMA cycle.

4. CP-06 Alternate Storage Site – As the tabletop exercise is performed, the ISCP is reviewed and discussed to ensure that information about backup procedures and an alternate storage site is identified and included in the plan. If backup procedures or alternate storage sites are not in place, a summary finding is annotated on the Checklist to document this issue.

5. CP-07 Alternate Processing Site – During tabletop exercises, the ISCP Test Plan must be reviewed to ensure an alternate processing site has been identified. If an alternate processing site has not been identified, a summary finding is annotated on the Checklist to document the issue.

6. CP-08 Telecommunication Services – Tabletop exercises for IRS IT systems and business applications not supported by IRS IT will include discussions about the telecommunication infrastructure and its DR capabilities, backup procedures, and validation that a DR plan exists for its recovery.

7. CP-09 Information System Backup – Discussions during tabletop exercises will focus on the ISCP to ensure that backup procedures are documented and implemented. The procedures must include information about the backup frequency, encryption of backup media, offsite storage, and timelines. If backup procedures have not been implemented, a summary finding is annotated on the Checklist to document this issue.

8. CP-10 Information System Recovery and Reconstitution – Tabletop discussions for this control will focus on the information in Section 5 of the ISCP to validate that procedures are in place to recover and reconstitute IRS IT systems and applications.

(6) Each BOD will be responsible for identifying a Data Collector who must document changes, issues, or findings identified during the tabletop exercise in the appropriate sections of the ISCP Testing Checklist. (IRS-defined)

(7) The ISCP Testing Checklist will be used as an artifact in TFIMS to document all tabletop exercises, functional exercises, and DR tests that are conducted. (IRS-defined)

(1) ISCPT utilizes the tabletop session as a training opportunity, as well as a testing exercise, to thoroughly understand the ISCP information and verify its accuracy and effectiveness. Training credit will be given to those that attend the entire tabletop exercise. (IRS-defined)

(2) Using the approved testing schedule, the assigned ISCPT Facilitator will send a calendar invitation for the tabletop exercise to all POCs thirty (30) calendar days prior to the day of the exercise. (IRS-defined)

(3) The assigned ISCPT Facilitator must provide the ISCP Testing Checklist with items 1-4 of the checklist pre-populated, along with the converted ISCP, to the Data Collector who is designated by the BOD Security PMO or application/system AO prior to the tabletop exercise. (IRS-defined)

(4) Using the converted version of the ISCP along with POA&M information stored in TFIMS the Data Collector must populate items 5 through 7 and Part A on the Checklist, prior to the tabletop exercise. If necessary, the Data Collector will communicate with appropriate BOD or IRS IT personnel to complete this task. The assigned ISCPT Facilitator must provide the ISCP Testing Checklist with items 1-4 of the checklist pre-populated to the Data Collector who is designated by the BOD Security PMO or application/system AO. (IRS-defined)

(5) Once the Data Collector has populated items 5 through 7 and Part A of the Checklist, the Data Collector must forward the Checklist and converted ISCP to all recipients, including the ISCPT Facilitator at least seven (7) business days prior to the tabletop exercise. (IRS-defined)

(6) During the tabletop exercise, the Data Collector is responsible for capturing on the Checklist all changes, observations, lessons learned, and summary findings that result from the tabletop discussions. The Date Exercise Completed block must be entered with the date the tabletop was performed. (IRS-defined)

1. At the start of the exercise, the Facilitator should welcome the participants to the event and request that the participants introduce themselves by name and give a general description of their roles as it relates to the application or system being tested. The Facilitator will discuss the scope of the exercise and logistical information. The Facilitator will walk the participants through the scenario and initiate a group discussion of the contents of the ISCP. The Facilitator may inject additional questions periodically for clarification purposes. The Data Collector documents issues to be included in the after action report. (NIST 800-84: Section 4.4)

2. Immediately following the facilitated discussion, the Facilitator and Data Collector should conduct an exercise debrief, in which they ask the participants in which areas they felt they excelled, in which areas they could use additional training, and which areas of the plan should be updated. (NIST 800-84: Section 4.4)

(7) After the exercise, the Data Collector has fifteen (15) work days to update the Checklist with the results of the exercise. The Facilitator will coordinate with the Data Collector as needed to provide guidance and to compare notes taken during the exercise. (IRS-defined)

1. The comments from the debrief, along with lessons learned during the exercise, must be captured on the checklist. The Checklist should include background information about the exercise, documented observations made by the Facilitator and Data Collector, and recommendations for updates to the ISCP for the application or system that was exercised. (NIST 800-84: Section 4.5)

2. Following the development of the of the Checklist after the tabletop exercise, the Data Collector may assign action items to select personnel to update the ISCP that was exercised. (NIST 800-84: Section 4.5)

(8) Once the post-tabletop Checklist update is completed, the Data Collector must send it to the ISCPT group mailbox at *IT IT DR Mailbox and the BOD Security PMO. ISCPT personnel must forward it to the assigned ISCPT Facilitator, who must review the Checklist to ensure that all information has been recorded. If Checklist corrections are needed, the Facilitator will coordinate with the Data Collector to ensure that the modifications are made. (IRS-defined)

(9) The ISCP Facilitator will ensure that all required testing has been completed for the application or system named on the Checklist. This may mean that the Facilitator will hold the Checklist until all additional testing has been completed and can be documented in Part B or Part C of the Checklist. If no other testing is required, the ISCPT Facilitator must prepare and send the Checklist back to the Data Collector and the BOD Security PMO within seven (7) business days for AO or AODR digital signature. (IRS-defined)

(10) The AO or AO Designee has 15 work days to sign the Checklist. (IRS-defined)

(11) ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡

(12) ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡

(13) ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡

(1) Functional exercises are performed by IRS IT personnel or by the BOD’s information system personnel when the application is not supported by IRS IT. Functional exercises are completed on backed-up data or information to prove back-up resiliency. (Refer to Exhibit 10.8.62-1.) (IRS-defined)

(2) Refer to Exhibit 10.8.62-2, ISCP Functional Exercise Methodology and Procedures. This exhibit provides step-by-step procedures for a backup retrieval and sampling pull for functional exercise activities. All functional exercises will be conducted using the approved procedures in Exhibit 10.8.62-2. (IRS-defined)

(3) As the production environment implements new technologies, strategies, and procedures, IRS IT and SRM must assess when to modify Exhibit 10.8.62-2 procedures to ensure that functional exercises can be performed to accommodate the updated production environment. (IRS-defined)

(4) During the functional exercise,

1. Functional exercises should prompt participants to carry out their roles and responsibilities as realistically as possible. Data Management from EOps or BOD IT personnel will conduct functional exercises. (NIST 800-84: Section 5.4)

2. The IRS IT or BOD information system personnel must take screen prints while the test is being conducted to submit to ISCPT as evidence. Evidence of backup recovery capabilities, includes, but is not limited to: (IRS-defined)
   - A screenshot that includes the name of the production server and the time the testing began.
   - If data is backed-up to media, evidence in the form of routing sheets, logs, or e-mail requests proving the length of time needed between the request for backup media from offsite storage and the receipt of that media at the test site.
   - If data is replicated, evidence that the data size from the production server matches the data size from the test site after the data is replicated.
   - Evidence that backup media and/or replicated data is readable.

3. The exercise director announces the conclusion of the exercise. Immediately following the exercise, the exercise director, controllers, and data collectors conduct an exercise debrief with the participants, requesting feedback from everyone present. (NIST 800-84: Section 5.4)

(5) IRS IT or BOD information system personnel must also provide evidence to validate that documented backup procedures in the ISCP are in place and that the ISCP includes information about the backup frequency, encryption of backup media, offsite storage site, and timelines for receipt of backup media from offsite storage during normal working hours and after hours. (IRS-defined)

(6) If no documented procedures are in the ISCP describing the backup process, the issue will be documented in Part B of the Checklist. In addition, if the backed-up data or information cannot be successfully retrieved or read, or if evidence cannot be captured for the exercise, these issues will also be documented in Part B of the Checklist. (IRS-defined)

(7) ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡

(8) Once the final checklist has been prepared for AO or AODR signature, ISCPT will send the checklist to the BOD Security PMO or Data Collector.(IRS-defined)

(9) ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡ ≡

(10) Upon receipt of the signed Checklist and supporting documentation from the AO or AODR, ISCPT must upload the Checklist into TFIMS as the validated artifact along with all supporting documentation. (IRS-defined)

(1) IRS is required to perform DR tests on all applications with a FIPS 199 High availability categorization, on assets that support any of the MEFs and on CIP assets. (IRS-defined)

(2) DR tests involve activities such as performing cutovers from one platform or system to another, relocation of systems/applications, or recovery of platforms and their hosted applications. As DR tests are performed on systems, sites, or platforms, hosted applications can benefit from these tests through coordination of the application ISCP review and the DR test activities. (IRS-defined)

(3) IRS or BOD IT personnel perform DR tests. During the performance of the DR Test, IRS or BOD IT personnel must complete the ISCP Testing Checklist Part C, and Test Case templates as they conduct the test. (Refer to Exhibit 10.8.62-1.) (IRS-defined)

(4) The ISCPT Staff will coordinate with IRS IT or BOD information system personnel to identify components, systems, and/or comprehensive tests to be planned based on FISMA, Treasury, and NIST requirements, and IRS executive-level priorities. (IRS-defined)

(5) Successful recovery of a production location at the alternate processing site is considered meeting the IRS DR Testing requirement if documentation and evidence are gathered. The Service may also consider combining tests with planned operational activities, such as restoring a backup, moving a server from one room to another, upgrading or patching operating systems or applications, or changing hardware components (e.g., swapping hard drives, replacing a failed power supply etc.). The results of this collaboration will define the scope and objectives for the tests. (IRS-defined)

(6) The ISCPT Staff will collaborate with designated BOD POCs to determine if the tests identified in collaboration with IRS IT or BOD IT personnel are compatible with the priorities and processing timeframes of the Business Unit. ISCPT and/or IRS IT information system personnel will coordinate with BODs to determine the level of involvement required from the BOD POCs. (IRS-defined)

(7) The ISCPT will collaborate with IRS IT or BOD information system personnel to create a DR test schedule annually based on IRS and FISMA requirements, FISMA timeframes, and business processing priorities. (IRS-defined)

(8) The ISCPT Staff will coordinate activities with IRS IT or BOD information system personnel to ensure that the ISCP Testing Checklist, Summary Report, and all testing documentation is completed before, during, and after testing. (IRS-defined)

(9) For Enterprise Computing Center or DR tests performed by BOD IT information system, IRS IT or BOD information system personnel will: (IRS-defined)

1. Coordinate with the designated IRS IT or BOD POCs to ensure the Test Case Template has been populated with pertinent information about the test such as scope detail, objectives, recovery personnel, support personnel, and test activities planned.

2. Ensure that IRS IT or BOD information system personnel identify the files needed to be transmitted in preparation for the tests and determine the date for transmission of data via IRS approved protocols.

3. Coordinate with Enterprise Computing Center (ECC) Security Management Office (SMO) personnel to reserve a conference room to hold meetings before, during, and after planned test activities as needed.

4. Coordinate with stakeholders to ensure that pre-test activities are completed.

5. Facilitate the creation of procedures to terminate the test in case operational issues necessitate it.

6. Coordinate with IRS IT and BOD POCs to ensure that all test participants including end users are familiar with the test termination procedures.

7. Coordinate with IRS IT or BOD information system personnel and POCs to ensure that end users are not adversely affected during planned test activities.

8. Conduct the DR test. (NIST 800-84: Section 6.4)
   i. The locations for tests vary based on the type of test being conducted and the test’s scope.
   ii. During a test, the mission of the organization should not be disrupted to the extent that the organization can no longer function and provide the services that it was created to provide. If there is any sign of a possible catastrophic disruption, or the safety of an individual is at stake or the security of the organization or its data is in question, the test director and any other member of the test staff should have the ability to terminate the test immediately.
   iii. After the test concludes, the test director should conduct an informal test debrief, requesting feedback from everyone present.

9. Coordinate with IRS IT organization or BOD POCs at the end of the test to ensure that test deactivation procedures are completed.

10. Review and evaluate the completed Test Case Template, worksheets, findings, corrective actions, and all test evidentiary documentation.

11. Comments from the debrief, documented observations made by the exercise staff, and recommendations made during the DR test, should be captured in an after action report. Managers may need to be briefed on DR test results. (NIST 800-84: Section 6.6)

12. Populate a test Summary Report to include findings, corrective actions, lessons learned, and summarize test worksheet results.

13. Facilitate post test meetings as needed to go over Summary Report, lessons learned, and corrective actions.

(1) For the purposes of reporting on the progress of exercises and testing, ISCPT must maintain a scorecard to document the progress of the ISCP tabletop and functional exercises and the status of the DR tests. (IRS-defined)

(1) ISCPT must upload the updated ISCP, signed Checklist, and all evidence and documentation for each ISCP tested application or system into TFIMS in a timely manner. ISCPT will also update the Contingency Planning (CP) fields in TFIMS with the appropriate dates. (IRS-defined)

(2) The following ISCP Testing documentation is uploaded into TFIMS after exercise/testing is completed: (IRS-defined)

• Updated Contingency Plan (ISCP)

• ISCP Testing Checklist

• Evidence collected from functional and/or DR tests

(3) The following Contingency Planning (CP) fields will be updated in TFIMS after exercise/testing is completed: (IRS-defined)

• Last CP Test Date (date all tests were completed).

• Next CP Test Date (one year from CP Test Date)