Developmental Test, Evaluation, and Assessments
Test & Evaluation Initiatives

Objective: Provide a coordinated framework for obtaining M&S capabilities and resources to enhance M&S procedures for early testing and prototyping efforts across the Adaptive Acquisition Framework pathways.

Current Effort: Address the following:

• Policy and guidance updates (e.g. M&S validation guidance and best practices)
• Assessment of model sufficiency to inform Test and Evaluation (T&E) (including cybersecurity and machine learning/artificial intelligence considerations)
• Adequacy of statistical engineering techniques for M&S Verification, Validation, and Accreditation (VV&A)
• Relevance of M&S in decision support frameworks
• Digital engineering strategies for M&S in T&E
• Resources required to train the workforce on these concepts

Collaboration agencies or communities: JHU/APL, STAT COE, IDA, EP&S, TRMC, DOT&E

Objective: Advance DE in enabling streamlined, automated, model‐centric T&E processes in a DE ecosystem through improved guidance, tools, and collaboration to support implementation of DE concepts and methodologies.

Current Effort: In collaboration with USD(R&E) stakeholders (e.g., Engineering Policy and Systems (EP&S)), Services, OSD, and industry, develop and promulgate tool selection methodology, facilitate Digital Engineering to support T&E workshops, define a digital Test and Evaluation Master Plan (TEMP) for developmental test, and support development of the T&E authoritative source of truth via the Digital Engineering Body of Knowledge.

• Build a comparison of policy and guidance between DoD and National Institute of Standards and Technology (NIST) to identify gaps, similarities, or areas of conflict; monitor and select applications of NIST investments

Collaboration agencies or communities: NIST, Services, OSD, JHU/APL, STAT COE, IDA, EP&S, TRMC, DOT&E

Objective: Identify key "Shift Left" practices and policy modifications for early and efficient testing, enabling programs to maximize test data as soon as possible and inform program development decisions.

Current Effort:

• Hold workshop to introduce Shift Left and Increase Technical Rigor (Tech Rigor) Initiatives content and implementation recommendations to DTE&A Deputy Directors and staff
• Engage with programs identified in consultation with DTE&A to implement best practices identified in IDA D-22771 (Shift Left) and D-22772 (Tech Rigor)
• Use the T&E Companion Guidebook as the primary source of T&E practices for programs (Collaboration with DOT&E)

Desired Outcome: (1) Early understanding of emerging requirements and technical performance measures to improve system substantiation and T&E support and (2) increased engineering rigor and early DT&E considerations.

Objective: Develop T&E tools and techniques for AI-enabled defense systems to be matured and more widely applied across DoD; advance T&E of AI-enabled systems to rapidly field critical capabilities, fully informed of their risks and rewards.

Current Effort:

• In collaboration with the T&E community and USD(R&E) stakeholders, produce a T&E framework - Future State for AI that shows internal and external alignment of activities and efforts related to AI/autonomy and T&E to help ensure DTE&A is focused on the right strategic questions.
• Host and participate in workshops and Technical Exchange Meetings (TEMs) to increase understanding of best practices and to drive future initiatives to better enable DoD to assess AI-enabled systems and autonomous systems
• Develop a white paper to define specific T&E evidence needed for AI-enabled systems across the broader software lifecycle

Collaboration agencies or communities: DOT&E, TRMC, JAIC, Services, IDA, STAT COE, USD(R&E) PD AI, USD(R&E) PD Autonomy

Objective: Transform how the DoD plans and conducts T&E to a more scientifically rigorous method to improve system behavior and combat mission capability data across the DoD to improve test execution, reduce program burden, and improve data-informed acquisition decisions.

Current Effort:

• Adapt the STAT process to create best practices for developing test strategies to support Independent Technology Readiness Assessments (ITRAs) and Service Technology Readiness Assessments (TRAs)
• Generate testing and analysis to better characterize emerging capabilities versus simple demonstrations under limited conditions allowing evaluators to more easily identify deficiencies and risk earlier in development, specifically:

  • Review current TRA practices across the services
  • Document data used at various stages to support TRA and identify gaps
  • Expand the STAT process to more effectively support TRAs
  • Write best practices for test strategy development to support TRA
  • Partner with pilot programs to implement best practices

Collaboration agencies or communities: DOT&E, TRMC, MITRE, 782TS/RWNG

Objective: Provide guidance to the T&E acquisition community regarding executions of DevSecOps (DSO) T&E within a DoD environment resulting in speed of delivery to warfighter and increased quality.

Current Effort:

• Continue collaborative review with EP&S and DOT&E representatives to complete version 1 of the DevSecOps T&E Companion Guide for Service T&E lead review and subsequent publication.

• Continue to research into Agile/DevSecOps T&E methodology in industry and academic documents to ensure the guide reflects state-of-the-art guidance
• Draw state-of-the-practice and lessons learned from programs of record using Agile/DevSecOps methods to capture actual experiences in the guide
• Collaborate with DOT&E to produce the next version of companion guide to ensure guidance is consistent throughout the OSD test community
• Collaborate with USD(A&S) and DoD CIO to make sure the DevSecOps software development guide is consistent with other DoD guidance
• Collaborate with DAU to ensure DevSecOps instruction incorporates the latest DevSecOps testing best practices and lessons learned from programs

Collaboration agencies or communities: EP&S, DOT&E, USD(A&S), DoD CIO, DAU

Objective: Mature the IDSK/Evaluation Framework (EF) and corresponding guidance to guide AAF programs' evaluation of operational and technical capabilities, drawing upon data from the full test continuum to inform acquisition and operational decisions.

Current Effort:

• Conduct IDSK/EF workshops to advocate for the methodology and refine the process
• Pilot updated process with AAF programs to use lessons learned, refine process
• Develop prototype IDSK/EF as a dashboard in Digital Engineering ecosystem and connect to test database (i.e., CHEETAS)
• Develop prototype IDSK/EF SysML-like model and connect to test database (e.g. TRMC CHEETAS) linking the IDSK-EF and Test Design Framework (TDF) with Digital Engineering tools and concepts
• Establish a program through which IDSK-EF (and Developmental Evaluation Framework (DEF)) subject matter experts will train DTE&A staff specialists on the IDSK-EF concept and the IDSK-EF Core Team facilitation process
• Collaborate with DOT&E to develop/update overarching policy and guidance updates (e.g., T&E strategy focus area chapter)

Collaboration agencies or communities: DOT&E, TRMC, EP&S, Service, T&E organizations

Objective: Provide a rigorous, repeatable, and systematic capability evaluation approach using Mission Engineering results combined with test and evaluation's (T&E) decision-supporting EF concept to inform Prototyping and Experimentation's (P&E) recommendations to operationalize in order to facilitate rapid modernization and transition into PoRs.

Current Effort:

• Identify a use case or real example (e.g., MITRE's Experimentation and Integration Cell) to exercise the decision support process focusing on tech transition into PoR
• Identify and implement pilots (e.g., MITRE's Experimentation and Integration Cell) to exercise the decision support process for S&T transition
• Participate and engage in the Rapid Defense Experimentation Reserve (RDER) action officer working groups (e.g., RDER transition, RDER studies, and mission threads) to support and/or identify opportunities to integrate DTE&A's capability evaluation approach as a proof of concept

Collaboration agencies or communities: DOT&E, TRMC, (USD) Mission Integration, USD(R&E) P&E

Objective: Expand consistent application of identify, deter, protect, detect, and respond.

Current Effort:

• PWN2H0NE: Facilitate the planning and conduct of the PWN2H0NE classified bug bounty event. Gather data to evaluate effectiveness, lessons learned, and to develop process improvements for PWN2H0NE events
• DoDM for Mission-Based Cyber Risk Assessments (MBCRA): Collaborative effort including JHU APL, IDA, and MITRE. Cross-walk stakeholder views and MBSE data requirements with the MBCRA standards to identify potential missed criteria. Effort will leverage information and artifacts generated from the Cyber Analytic landscape study

Collaboration agencies or communities:

• PWN2H0NE: USAF CROWS, NCRC/TRMC
• DoDM for MBCRA: JHU/APL, AF: Service T&E (DT/OT) Reps, Service Systems Security Engineering, Service AOs, OUSD A&S, DHS, DOT&E, DoD CIO, OPCA, NSA, R&E/RS, J6, MDA, NGC

Objective: Improve use, awareness of, maturity, and access to relevant T&E analytic tools.

Current Effort:

• OACRA: Deliver and pilot use cases for the ontology (CTTs, MBSE analytics)
• Tool Integration: Produce prototype integration between a live emulation tool (CALDERA) and an existing analytic ontology (OACRA). Provide a mapping between CALDERA objects and actions to OACRA. Demonstrate a capability to convert CALDERA reports into machine- and human-readable formats with OACRA mappings. Help identify areas of interoperability to enable compatibility in the products of the tools to inform analysis
• Cyber Analytic Landscape (CAL): Conduct a small number of workshops focused on cyber analytic questions, data needs, and MBSE profiles. Examine current MBSE approaches for cyber tools and develop a more general and sustainable approach. Inventory cyber stakeholders, stakeholder decisions and questions, supporting analytics, and data consumed by the analytics

Collaboration agencies or communities:

• OACRA/Tool Integration: TRMC, Services, MDA
• CAL: MITRE, IDA, Cross DoD/OSD

Objective: Drive cyber requirements integration into early detailed design through improved capability to test security, survivability, and resilience with contractor, functional, and government test organizations.

Current Effort:

• Cyber T&E Companion Guide: Improve contractor accountability for testable, measurable, and achievable cyber requirements throughout the program to verify networks, systems, and information are survivable and resilient to cyber threats. Expand processes, technical capability for integrated cyber testing, supply chain and program protection. Expand and improve T&E processes, strategies, and implementation leveraging guidance, best practices and technology to accelerate the survivability and operational resilience of warfighting capabilities to cyber threats
• Cyber T&E Credential: Improve workforce knowledge, skills, and abilities in cyber T&E processes, tools, technology and operational mission relevant outcomes
• DOE for Cyber: Work with acquisition programs to develop, plan, and execute cyber test plans in such a way to prevent (1) Inefficient use of cyber test resources and (2) Increased risk of unknown/misunderstood cyber vulnerabilities.

Collaboration agencies or communities:

• Cyber T&E Companion Guide: IDA, JHU/APL, DOT&E with feedback from Services, Agencies, Cross-OSDA, NSA, JS J6;
• Cyber T&E Credential: EP&S, DAU
• DOE for Cyber: STAT COE, acquisition program personnel, DTE&A staff specialists