Increasing Federal Cybersecurity and Privacy Program Maturity: Conducting a Program and Operations Review

Criterion | Cybersecurity CoE

Increasing Federal Cybersecurity and Privacy Program Maturity: Conducting a Program and Operations Review


By Dan Chandler, CISO and Bob Heckman, CIO, Criterion Systems, Inc.

We’ve written in the past about how improving Federal Cybersecurity and Privacy (CS&P) programs demand a comprehensive cybersecurity maturity model such as the FY 2018 Inspector General FISMA Reporting Metrics v1.0. Federal organizations can then use this model to define a flexible roadmap for maturing their CS&P program. Creating this roadmap requires the organization to have an accurate view of its as-is state. Achieving this view, however, is easier said than done, as it needs to incorporate applicable FISMA, OMB, NIST, DHS and OIG/CIGIE requirements, criteria, and guidance. One word of caution is that the current OIG/CIGIE model does not contain maturity criteria for each subcategory level of the NIST Cybersecurity Framework. In response, Criterion has created a Cybersecurity Program and Operations Review format mapped to its cybersecurity maturity model. This format reflects the various Federal requirements mentioned above as well as our experience in where CS&P programs run into difficulty, incorporating best practices and lessons learned derived from numerous NIST Special Publications (SP) and other applicable recommendations and guidelines.

Criterion’s review, therefore, not only gives a snapshot of where a federal organization is in their CS&P program implementation, it also identifies existing program weaknesses, gaps, vulnerabilities, and deficiencies/discrepancies, and provides appropriate mitigation strategies to correct and remediate the findings and strengthen the other CS&P programs. Based on the review results, Federal organizational management will be able to make well-informed risk-based decisions regarding CS&P-related initiatives, mitigation strategies, and remediation activities.

Five-Phase Review Process

Criterion’s Cybersecurity Center of Excellence (CoE) conducts program and operations reviews in five phases designed to evaluate the overall management, operational, and technical readiness of the Federal organizational CS&P program or project:

  1. Phase I, Program and Technical Assessment: Data is collected through documentation reviews and targeted interviews. Applicable CS&P-related documentation includes policies, plans, processes, procedures, previous risk assessments, etc. Other inputs include the Federal organizational and CIO strategic/tactical plan, top Federal organizational risks, and new information system development. The review team holds interactive stakeholder meetings and uses techniques such as examination, test, inspection, observation, inquiry, confirmation, analysis, and discussion to achieve the review’s objectives.
  2. Phase 2, Program and Technical Review: In this phase, the data collected in Phase 1 is entered, scored quality checked. We then review, analyze, and evaluate the results to identify prioritized areas of improvement based upon current and anticipated performance. The focus of our initial review and evaluation in this phase is to establish a baseline CS&P framework profile or risk level and a CS&P maturity level for each functional program.
  3. Phase 3, Program and Technical Baselining: We develop and document the initial set of CS&P findings including mitigation strategies, options, and recommendations. These findings are each categorized as a potential vulnerability, weakness, gap, or deficiency/discrepancy that would result in accidental or intentional unauthorized modification, disclosure, destruction, and denial of service to the information or system throughout the entire life cycle. We quickly notify the Federal organizations if the review reveals any critical finding or event that requires their immediate intervention.
  4. Phase 4, Client Debriefing: We prioritize our mitigation strategies, options, and recommendations for keeping ahead of advancing and evolving cyber threats and present them to the client. Upon client concurrence, we develop the draft report along with more detailed roadmaps, corrective action plans and/or project plans that lead to the required, improved, or additional CS&P capabilities.
  5. Phase 5, Remediation: We conduct and/or monitor remediation efforts; collect applicable artifacts/evidence; conduct and document applicable review and/or test activities to verify/validate remediation results; modify or update roadmaps, corrective action plans, and/or projects plans; and update Plans of Actions and Milestones (POA&Ms) as required.

Review Challenges

Although critical information can be gathered from a thorough examination of the CS&P program documentation, important informational gaps relevant to the CS&P program can still exist due to disconnects between how the program is implemented and followed, changes in the operational environment and/or situations, and/or inadequacies in the documentation. To close these gaps, we tailor and customize our maturity model tool to incorporate Federal organizational input. For example, it includes more than 1700 indicators based upon input from FISMA, OMB, NIST, DHS, OIG/CIGIE and industry best practices. Additional review questions are also added to focus on critical items of interest to the organization.

Another key challenge is the scoring of the data. Criterion’s maturity model brings together and/or rationalizes the various guidelines, measures, and taxonomies into one place. This enables an easier scoring or weighting of the data gathered into three classifications that, when reviewed together, can determine the applicable CS&P maturity level: Core Functions, Categories, and Subcategories. Subcategories are ranked on a scale of 0-4 (0 being the lowest ranking) to determine the Framework Profile or risk level of each category. Also, the subcategories are ranked on a scale of 1-5, with 1 being the lowest, to determine the maturity level of each category. Both ratings are weighted based upon Federal organizational input and aggregated to produce a weighted average for their respective category. Similarly, the category rankings are then aggregated to yield a weighted average for each heading. This bottom-up approach produces ratings with three levels of detail, and an overall rating that provides the CS&P Framework Profile or risk level and the CS&P Maturity level score for the whole program.

Criterion’s Cybersecurity Program and Operations Review process, which feeds into its cybersecurity maturity model, helps promote consistent and comparable metrics and criteria in the CIO and OIG metrics processes (FISMA reporting) while providing the Federal organizations with a meaningful independent assessment of the effectives of their CS&P program. Once identified, the existing and desired CS&P program maturation points provide the basis for the team to develop, refine, and document a roadmap of manageable sub-projects that build upon one another to evolve Federal organizations toward optimized CS&P practices and compliance.

Other articles in our “Increasing Federal Cybersecurity and Privacy Program Maturity” series:

Learn more about Criterion’s CyberScale™ Compliance and Risk Management Solution here: