\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_memo_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{Automated Testing}
\vspace{0.1in}
\labdate{January 2022}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
%\labtoc{small}
\end{labcover}

\wikidoc{file=Memo.VVIntro}
\wikidoc{auth=mikerb@mit.edu}
\wikidoc{toc=false}

%==================================================================
\section{Overview}
%==================================================================

Testing through simulation has always been used as a way of verifying
correctness of autonomy algorithms in robotics. The conventional
thinking is that for every hour of real-world robot deployment, there
may be 100 to 1000 hours of simulation. The actual ratio depends on
many factors but is perhaps driven most by the cost of the robot, the
risk of losing the robot or harming people or property, and the cost
of time on a test range, or research vessel.

\vspace{0.1in}
\noindent
In early prototype development, new autonomy code moves very quickly
onto a robot. Sometimes code changes are even made directly on the
robot itself during pauses in field testing. The opposite is true as a
robot becomes a system with urgent needs for reliability and safety.
Software may be "locked down" days or months prior to a major field
trial or delivery to a user. At this point the software complexity is
also higher - more lines of code, more developers involved, and more
interplay between complex components. The bar is much higher for
accepting a new software version. The time and cost of the acceptance
testing process is a major factor in deciding to upgrade the autonomy
software to a new release.

\vspace{0.1in}
\noindent
Lowering the cost (shortening the timeline) for acceptance testing is
a major factor in ensuring a high rate of innovation in autonomous
systems. Robots and robot systems cannot evolve if their autonomy
software is frozen. Developers cannot innovate if their newest code
does not reach the robots. There are several methods for accelerating
acceptance testing through simulation. They include:

\begin{packed_itemize}
\item {\em Faster-than-realtime simulation}: A simulation should
  proceed as fast as the computer(s) allow. A simulation involves
  multiple processes working together and the ability to run faster
  than real-time is usually limited by the slowest or most CPU
  intensive process. Higher-fidelity simulation applications may in
  some cases be swapped out for lower fidelity applications simply to
  improve the overall rate of simulation.
  
\item {\em Automated testing}: If a simulation can be automated, this
  can mitigate slower running simulations since they can be left
  unattended, and presumably sequenced into a batch of simulations run
  continuously overnight or the weekend.

\item {\em Distributed simulation}: If large numbers of computers are
  available, even if containerized systems on cloud services, this also
  can mitigate slower simulations. Presumably N computers running real-time
  simulations is about the same as one computer running N times real time.
  (But even better to have N times real time, on N computers!)

\item {\em Distributed multi-vehicle simulation}: If the simulation
  involves multiple vehicles, say for collision avoidance testing, or
  swarm simulations, even a simulation that can run 20x realtime, per
  vehicle, may only be able to handle 2x realtime if simulating 10
  vehicles. In some cases an N vehicle simulation just may not be
  possible on a single computer.
  Distributed multi-vehicle simulation involves multiple
  computers participating in the same single simulation, typically
  one computer per simulated vehicle.
  
\item {\em Randomization of testing}: Batch testing of simulations may
  involve a single mission type with input parameters varied in ways
  to test the robustness of autonomy configuration parameters,
  starting vehicle positions, or starting positions of contacts or
  obstacles. Starting configurations may be enumerated explicitly or
  randomly generated. Tools for random parameter generation need to
  support more than just random number selection. In the case of
  vehicle and obstacle generation, the tools need to avoid generation
  of starting conditions that are doomed to fail, such as vehicles on
  top of, or near each other, or obstacles. In the case of distributed
  multi-vehicle testing, randomization tools need to be implemented in
  a decentralized manner.

\item {\em Automated analysis tools}: Simulations vary in what they
  are testing. In certain simulations, a simple pass/fail condition is
  tested. In other simulations, events of interest are happening
  continuously and the analysis steps are non-trivial. In these cases
  specialized apps need to be developed to run alongside the primary
  mission applications, to enable an automation of the mission
  testing.

\item {\em Post-Mission replay tools}: When or if missions do fail
  their tests, post-mission tools are essential for analyzing what may
  have gone wrong and fixing the problem. While technically not part
  of the automated mission testing pipeline, when all tests are passing,
  the post-mission tools are essential in practice fixing problems and
  tuning parameters.
\end{packed_itemize}

\vspace{0.1in}
\noindent
This document describes the simulation and automated simulation environment
for the MOOS-IvP autonomy software environment. It will cover (a) general
mission structure, (b) mission launch structure, (c) tools for preparing
missions, (d) tools for analyzing ongoing missions, (e) tools for post-mission
analysis and debugging, (f) data visualization methods, and (g) the structure
for daily automation and reporting of mission verification results.

\end{document}



