Types of mission testing
=======================================================
(a) A human operator launches a mission, observes the activity, and
    after some period of time decides the mission has provided enough
    information. The user halts the mission and rates the result.

    Launch (Human)
    Halt (Human)
    Assess (Human)

(b) A human operator launches, auto-completes, auto-analysis

    Launch (Human)
    Halt (Automatic)
    Assess (Automatic)

(c) A human operator launches a script containing several fully automated
    test

    Batch Launch (Human)
      Launch (Automatic)
      Halt (Automatic)
      Assess (Automatic)
    Batch Assessment (Human



Types of Analysis
======================================================

Single Pass Fail 

  - Did an event happen when it was expected

Perpetual Single Pass/Fail

  - Confirming an event that never should happen, never happened
    (e.g., multiple obstacle avoidance over time etc)

Continuous Pass/Fail Counter

  - Checking a condition that will fail periodically and we are
  checking the rate of failure. Other data may be stored with each
  case. For example, did ownship "catch" an intruder? Also recording
  the intruder's average speed

Countinuous Evaluation

- Run until a certain criteria has been satisfied, e.g., endflag
  or max amount of time: then post a variable representing an
  evaluation



Comparison to sister mission


Types of Missions
====================================
o Single Vehicle
o Multi-Vehicle, but focus on on rating a single ownship
o Multi-Vehicle, rating is based on group performance



Types of Tests
=======================================================

A Test
  - launch
  - halt
  - assess

A Batch Test
  - A set of tests
  - Batch results assessed
  

Exhaustive Batch Test
  - Starting positions exhaustively tested
  - Autonomy parameters exhaustively tested
  - Sensor parameters 

Parameter Randomized Batch Test
  - Starting positions randomized
  - Autonomy parameters randomized
  - Sensor parameter
  
Natural Randomized Batch Test
  - Hold starting positions, autonomy params and sensor params
  - Let natural system randomness vary




When to Measure/Test

Automated mission testing implies that the launch of the mission is 


Two types of 





Types of evaluations


1. Simple pass-fail logic evaluations

   o The result will be a Boolean
   o There is a single pass/fail check after a particular event

   

   Examples:

   A. Check that an endflag was posted at the end of a waypoint behavior

   
    


   This is typically to check that a certain kind of stated
   functionality, for a particular app or particular behavior,
   performs as state.


2. Complex pass-fail logic conditions

   o The result will be a Boolean
   o Multiple pass/fail checks are performed after multiple events

   Examples:
   o 

3. o The result will be a Boolean
   o A vehicle runs for some period of time