Verification and Validation (V&V) of System Behavior Specifications


Principal Investigator:  Dr. Kristin Giammarco, Naval Postgraduate School (kmgiamma@nps.edu)

Timeframe:  November 2016 – September 2018

Category: All

Sponsor: NAVAIR


Description

Objective: The objectives of this work are aligned to NAVAIR Systems Engineering Transformation (SET) tasks as follows:

Model-based V&V Demo

  • Formalize UAV behavior specifications into MBSE architecture tool(s)
    •  using Monterey Phoenix for comprehensive use case scenario generation
    • Core Value Proposition: Manual drawing of a limited set of use cases is replaced with automatic and comprehensive scenario generation, enabling humans to spend more time on requirements analysis and V&V tasks that cannot be automated. Correction of errors in system behavior is then done much earlier.
  • Demonstrate use of the UAV behavior model for early V&V analysis of requirements
    • using MP to expose positive and negative system behaviors permitted by the design
    • Core Value Proposition: Requirements gaps are identified and fixed early (before contracting), through inspection of a comprehensive set of use case scenarios.Formalize patterns of common design flaws or other model properties
  • Formalize patterns of common design flaws or other model properties
    • using MP event grammar to store system behavior templates
    • Core Value Propositions: Save money by eliminating error-prone, labor-intensive, and expensive manual checking, and provide specification for testing contracted models for presence or absence of wanted and unwanted system behaviors discovered during early model-based V&V.

Training Content Development and Delivery

  • Work on this task is planned in FY18.

Coaching & Mentoring

  • Create a catalog of typical architecture model views for behavior containing good practices (patterns[1]), poor practices (anti-patterns[2]), and pattern / anti-pattern examples.
  • Provide ongoing mentoring and coaching support as needed on pilot projects.

Approach:

  1. Prepare source data / conduct pre-work
  2. Develop Design Reference Mission
  3. Conduct MP behavior model development
  4. Conduct functional requirements behavior model-based V&V
  5. Conduct crosswalk of MP, SysML and LML for model transformation effort
  6. Conduct nonfunctional / system-wide requirements behavior model-based V&V
  7. Transition the MP Analyzer tool to the workforce

[1] Tool-agnostic patterns with style guidance for certain critical and useful model views will be incorporated.

[2] Discovered practices that should be avoided are often called anti-patterns since they illustrate examples contrary to good practice.

Deliverables

Major Deliverables:

  • MP Analyzer tool and accompanying documentation for transition (NPS, November 2016)
  • Innoslate model containing any imported requirements and corresponding UAV surrogate models (NPS, September 2017)
  • MP model with corresponding exhaustive set of use case variants based on provided surrogate models (NPS, September 2017)
  • Presentation with results of V&V requirements analysis activity on the MP model and corresponding use case variants (NPS, September 2017)
  • Catalog of discovered architecture patterns that are reusable on other NAVAIR systems that use MCSE MPTs (NPS, September 2017 v1, September 2018 v2)
  • Technical report and presentation containing findings, recommendations, and instructions for applying demonstrated MCSE MPTs on other systems (NPS, September 2017 v1, September 2018 v2)
  • Presentation assessing model transformation potential between UPIA and MP (Stevens Institute, September 2017)
  • Working implementation of the MP Analyzer in the SERC Laboratory (Stevens Institute, September 2018)
  • Zip of .mp files
  • Interim Technical Report – September 30, 2017

 

Publications

Publications

Giammarco, Kristin and Katy Giles. “Verification and Validation of Behavior Models using Lightweight Formal Methods.” Proceedings of the 15th Annual Conference on Systems Engineering Research (CSER), Redondo Beach, CA, March 23-25, 2017.  Received Best Paper Award for Transition in Systems Engineering Research sponsored by MITRE.

Giammarco, Kristin and Clifford A. Whitcomb. “Comprehensive use case scenario generation: An approach and template for modeling system of systems behaviors.” Proceedings of the 12th Annual System of Systems Engineering Conference, Waikoloa, HI, June 18-21, 2017.

Giammarco, Kristin. “Practical Modeling Concepts for Engineering Emergence in Systems of Systems.” Proceedings of the 12th Annual System of Systems Engineering Conference, Waikoloa, HI, June 18-21, 2017.

Giammarco, Kristin. “Architecture Modeling Software Analytics: Model quality and maturity assessment using automated tools.” Proceedings of the 12th Annual System of Systems Engineering Conference, Waikoloa, HI, June 18-21, 2017.

Quartuccio, John, Kristin Giammarco, and Mikhail Auguston.  “Identifying Decision Patterns Using Monterey Phoenix.” Proceedings of the 12th Annual System of Systems Engineering Conference, Waikoloa, HI, June 18-21, 2017.

Quartuccio, John, Kristin Giammarco, and Mikhail Auguston.  “Deriving Stochastic Properties from Behavior Models Defined by Monterey Phoenix.” Proceedings of the 12th Annual System of Systems Engineering Conference, Waikoloa, HI, June 18-21, 2017.

Giammarco, Kristin. “Verification and Validation of Behavior Models using Lightweight Formal Methods.” Invited Talk, SOSECIE Webinar, August 8, 2017.

Giammarco, Kristin, Ron Carlson, Mark Blackburn, Mikhail Auguston, Rama Gehris, Marianna Jones, John Quartuccio, Kathleen Giles, Chris Wolfgeher, and Gary Parker.  “Verification and Validation (V&V) of System Behavior Specifications.” Interim Technical Report SERC-2017-TR-0176.  Hoboken, NJ: Systems Engineering Research Center.  September 30, 2017.

Research Team

Researchers

  • Kristin Giammarco, Naval Postgraduate School
  • Mark Blackburn, Stevens Institute of Technology
  • Ron Carlson, Naval Postgraduate School
  • Mikhail Auguston, Naval Postgraduate School
  • Rama Gehris, Naval Postgraduate School
  • Marianna Jones, Naval Postgraduate School
  • John Quartuccio, Naval Postgraduate School
  • Kathleen Giles, Naval Postgraduate School
  • Chris Wolfgeher, Naval Postgraduate School
  • Gary Parker, Naval Postgraduate School

Collaborating Institutions

Project Researchers