<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{pMissionEval: Evaluating User-Defined Mission Success}
\vspace{0.1in}
\labdate{January 2025}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labsrc{project-pavlab/appdocs/app\_pmissioneval} 
\labtoc{small}
\end{labcover}

\wikidoc{file=IvPTools.PMissionEval}
\wikidoc{bugz=pMissionEval}
\wikidoc{auth=mikerb@mit.edu}

%=================================================================
\section{Overview}
%=================================================================
\label{sec_pme}

The \app{pMissionEval} application is a tool primarily used in a
mission that is part of an automated test. An automated test is
typically run headlessly, i.e., no GUI, no human interaction. It is
meant to at times complement other MOOS apps that serve to evaluate a
mission.  For example, in an obstacle avoidance mission, there may be
other another app that detects collisions, and another app that
randomizes aspects of the obstacle placement. The role of
\app{pMissionEval} is to work in conjunction with the apps of the
mission to (a) define when a mission has completed, (b) define what
constitutes the result to be reported, and (c) write the mission
salient mission configuration aspects and results to a file in a
user defined format.

The typical topology is shown below, although it is possible to run
\app{pMissionEval} in a vehicle commmunity, or in a simple Alpha-like
mission with a single MOOS community.

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.6\textwidth]{figures/pme_topology.png}
  \caption{{\bf Typical pMissionEval Topology:} Typically run in
    the shoreside community, \app{pMissionEval}, performs a series
    of evaluations at events configured by the user, with results
    posted either to the MOOSDB or to a file, or both.}
\label{fig_pap_topology}
\end{figure}

\noindent
Note in the figure that two other apps are typically operated in
coordination:

\begin{packed_itemize}
\item \app{pAutoPoke}: When the mission is launched, this app will
  poke the MOODB with the commands normally provided by a human
  command to kick off a mission, e.g., \var{DEPLOY=true}.
\item \app{uMayFinish}: After the mission is launched, from within a
  script, this app is launched from with in the script, essentially
  blocking the script until the mission is finished. This app connect
  to the MOOSDB and monitor for a posting such as
  \var{MISSION\_EVALUATED=true}. Then disconnect from the MOOSDB and
  exit, presumably to allow the remainder of the blocked script to
  complete bring down the mission.  
\end{packed_itemize}

\noindent
The primary action produced by \app{pMissionEval} is the logging of a
single line of text in a named file, e.g., \var{results.txt}. This
file is named in a configuration parameter for \app{pMissionEval}.
After a mission has been completed, there may be a line produced like:

\vspace{0.1in}
\begin{fileverb}
  form=alpha, mmod=cycles, grade=pass, mhash=K-POSH-TIME, date=250126, time=1508
\end{fileverb}
\vspace{0.1in}

\noindent
When \app{pMissionEval} determines that evaluation has completed, it
will (a) write a result like like the one above to the results file,
and (b) post something like \mvar{MISSION\_EVALUATED=true}, to match
the configuration of \app{uMayFinish} to trigger the shut down of the
simulation. 

%=================================================================
\section{Four General Mission Types}
%=================================================================

Consider the following mision types:

\begin{packed_itemize}
\item A simple pass/fail sanity check on certain defined features of
  an app or a behavior. Sometimes referred to as a unit test.

\item A performance pass/fail test that requires a lengthy mission
  duration, with randomization introduced during the mission.
  Internally the pass/fail determination may be derived from a non
  Boolean score, e.g., the number of vehicle near-misses over 100
  obstacle encounters, but the mission is still a pass/fail test, with
  the test designer deciding whether success is say 0 near misses or
  some tolerable number.

\item A test that produces a performance score, as in the example
  above, also possibly involving randomized starting conditions, e.g.,
  obstacle locations. The difference is that this test also has
  several configuration variations, such as vehicle speed, turning
  capability, sensor distance, communications drop-out. Rather than
  this mission being a single component of a CI pipeline, it is part
  of set of sensitivity test, find the performance of algorithms
  relative to configuration and environment assumptions. A user
  may later wish to identify a few of the variations later to be
  part of a CI pipeline.
\item A competition mission that pits two teams, producing a resulting
  winner and score.   
\end{packed_itemize}

\vspace{0.1in}
\noindent
Each of these four mission types are fleshed about a bit more
in the following examples.

%=================================================================
\subsection{The Alpha Mission Pass/Fail Verifying Event Flags}
\label{sec_pme_alpha1}

The Alpha mission is the first hello-world mission to where new users
of MOOS-IvP are directed on an initial download. A vehicle starts and
proceeds through five waypoints in a home-plate pattern. It may
traverse the patterntwice before finishing and returning home.

\vspace{0.1in}
\noindent
The waypoint behavior has certain event flags defined. The
\bvar{endflag} is defined for all behaviors and is published when the
behavior "completes".  For the waypoint behavior, completion is when
it has traversed all waypoints.  If the waypoint specific parameter
\var{repeat} is set to 1, then the behavior will not complete until it
has traverse all five points twice.

\vspace{0.1in}
\noindent
The waypoint behavior has two additional event flags unique to this
behavior: the \bvar{cycle\_flag} and \bvar{wpt\_flag}. The former is
published each time the waypoint behavior completes a set of
waypoints, and the latter is published each time the behavior arrives
at any waypoint. So, for a waypoint behavior configured with five
waypoints and \bvar{repeat=1}, the \var{cycle\_flag} should be posted
twice, and the \bvar{wpt\_flag} should be posted 10 times.

\vspace{0.1in}
\noindent
For such a mission, something like the below result line would be
expected. 

\vspace{0.1in}
\begin{fileverb}
  form=alpha, mmod=eflags, grade=pass, mhash=J-BARE-LEAF, date=250126, time=1601
\end{fileverb}
\vspace{0.1in}

\noindent
The \var{form=alpha} indicates this is a result of the alpha mission.
The \var{mmod=eflags} is a "mission modification" tag. This is
discussed further in Section \ref{sec_pme_mmod}. A mission may have
multiple mods, each to test something slightly different, thus allowing
one mission folder to serve multiple purposes.

%=================================================================
\subsection{The ObAvoid Mission Pass/Fail Verifying Obstacle Avoidance}
\label{sec_pme_obavoid1}

In the ObAvoid mission, a vehicle is driven around a race-track like
pattern, through a rectangular region situated between the two ends of
the racetrack pattern. The region is populated with a randomly
generated obstacle field.  A shoreside MOOS app,
\app{uFldCollObDetect}, runs and montitors the position of vehicles
relative to the obstacles in the field. For each obstacle encounter,
the closest point of the encounter is noted, and evaluated. An
encounter with an obstacle, with a closest point of approach (CPA),
wwithin a certain distance, is considered a collision. A near-miss
might be defined as an encounter with a CPA distance within a slightly
larger distance.

\vspace{0.1in}
\noindent
The automated version of this mission can be configured to run the
vehicle for a given number of times around the racetrack, or for a
given amount of obstacle encounters. After a certain amount of runs,
the mission receives a passing grade if no collisions are detected.

\vspace{0.1in}
\noindent
For such a mission, something like the below result line would be
expected. 

\vspace{0.1in}
\begin{fileverb}
  form=obavoid, mmod=ob10, grade=pass, mhash=K-ICED-DINO, date=250126, time=1614
\end{fileverb}
\vspace{0.1in}

\noindent
The \var{form=obavoid} indicates this is a result from the obavoid
mission.  The \var{mmod=ob10} is a modifier that may indicate in this
mission there were ten randomly generated obstacles.


%=================================================================
\subsection{The ObAvoid Mission with Sensitivity Analysis}

In the same ObAvoid mission as above, the mission could be graded as a
\var{fail} if a collision with an obstacle is detected. I could also
be given a score, from 0 to 100, based on the percentage of
near-misses noted for each encounter with an obstacle.

\vspace{0.1in}
\noindent
The important distinction is that mission can be varied in several
ways. (a) the density of the obstacle field, (b) the minimal
guaranteed separation distance between randomly generate obstacles in
the field, (c) the range at which the vehicle can detect the obstacle,
(d) the turn rate of the vehicle.

\vspace{0.1in}
\noindent
An experiment would involve potentially large numbers automated
simulations. The result of each simulation would be a single pass/fail
grade or score. The goal is plot the relationship between the above
mentioned conditions, versus performance.

\vspace{0.1in}
\noindent
For such a mission, something like the below result lines would be
expected. 

\vspace{0.1in}
\begin{fileverb}
  form=obavoid, mmod=ob6, hits=0, nmisses=0, mhash=A-WEAK-BEER, date=250126, time=1614
  form=obavoid, mmod=ob6, hits=0, nmisses=0, mhash=K-LAZY-WEST, date=250126, time=1615
  form=obavoid, mmod=ob6, hits=0, nmisses=0, mhash=P-DEAD-ENZO, date=250126, time=1617
  ...
  form=obavoid, mmod=ob10, hits=0, nmisses=2, mhash=M-INKY-SIZE, date=250126, time=1623
  form=obavoid, mmod=ob10, hits=0, nmisses=0, mhash=D-COOL-REED, date=250126, time=1624
  form=obavoid, mmod=ob10, hits=0, nmisses=1, mhash=M-EASY-VICE, date=250126, time=1637
  ...
  form=obavoid, mmod=ob14, hits=0, nmisses=5, mhash=D-DANK-PHIL, date=250126, time=1639
  form=obavoid, mmod=ob14, hits=1, nmisses=8, mhash=G-COLD-KERR, date=250126, time=1641
  form=obavoid, mmod=ob14, hits=1, nmisses=2, mhash=C-MAIN-KNOX, date=250126, time=1644
  
\end{fileverb}
\vspace{0.1in}

\noindent
The \var{form=obavoid} indicates this is a result from the obavoid
mission.  The \var{mmod=ob10} is a modifier that may indicate in this
mission there were ten randomly generated obstacles. The overall test
involves several instances of each configuration. In this example, the
number of obstacles presumably increases the difficulty in obstacle
avoidance and the data should enable a plot relating obstacle density
with to the frequency of hits or near misses.

%=================================================================
\subsection{The Rescue Mission Head-to-Head Competion}

The Rescue mission is a head-to-head competition mission from the MIT
2.680 Marine Autonomy class. This mission is essentially in the style
of an Easter egg hunt. Two vehicles are deployed to rescue N swimmers,
by visiting each swimmer location. The vehicle to visit the majority
of the N swimmers wins. There is an game manager app running in the
shoreside community called \app{uFldRescueMgr}. This app will publish
something like

\vspace{0.1in}
\begin{consoleverb}
  WINNER   = blue
  SCORE    = blue=9 # green=7
  SWIMMERS = 17

  COMPETITION_COMPLETE = true
\end{consoleverb}
\vspace{0.1in}

\vspace{0.1in}
\noindent
When launched with a GUI, the participants could watch the mission
play out, and check the scoreboard in the appcasting output of
\app{uFldRescueMgr} to see the winner and indication of completeness.
In an automated headless mode, \app{pMissionEval} would wait for
\mvar{COMPETITION\_COMPLETED} to be true, and publish a line to the
results file looking something like: 

\vspace{0.1in}
\begin{fileverb}
  form=rescue, winner=blue, score=blue=9 # green=7, mhash=K-PLUS-MOTH, date=250126, time=1644
\end{fileverb}
\vspace{0.1in}

\vspace{0.1in}
\noindent
Since the competition involves randomly generated starting positions
and locations of swimmers, an experiment may involve many such
automated competition missions to get a good feel for the team with
the better strategy.




%=================================================================
\section{Basic Operation}
%=================================================================

In any mission, \app{pMissionEval} will do essentially the following
three things:

\begin{packed_itemize}
\item Wait until the proper point in the mission
\item Evaluate the results of the mission
\item Post the results to either a file or to the MOOSDB, or both
\end{packed_itemize}

\vspace{0.1in}
\noindent
In an automated, headless mission, there is always the concern about having
an absolute timeout, in case for some reason the first stage above fails
to happen. However, \app{pMissionEval} is typically not concerned about
this. The \app{uMayFinish} app is normally configured with an absolute
for aborting a mission. 

%=================================================================
\subsection{Conditions and Mission Evaluation}
\label{sec_pme_cond}

\app{pMissionEval} can be configured with one or more logic conditions
confiured to convey a mission stage or mission completeness. A simple
example is

\vspace{0.1in}
\noindent
\pvar{lead\_condition = ARRIVED = true}

\vspace{0.1in}
\noindent
When configured this way, \app{pMissionEval} will register for the MOOS
variable \mvar{ARRIVED}. When it receives mail with a value of \var{true},
the mission will be evaluated and results posted. Evaluation in terms
of pass/fail is determined by an additional two kinds of conditions,
{\em pass conditions} and {\em fail conditions}. A result of {\em pass}
will be award if (a) all pass conditions hold and (b) no fail condition
holds. For example, the above line may be further configured with:

\vspace{0.1in}
\noindent
\pvar{pass\_condition = ODOMETRY &lt; 1500} \\
\pvar{pass\_condition = FOUND\_OBJECT = true}  \\
\pvar{fail\_condition = COLLISIONS &gt; 0}  \\

\vspace{0.1in}
\noindent
The above three lines will be evaluated only when the lead condition
is satisfied, \mvar{ARRIVED=true}. If both pass conditions are
satisfied, and none of the fail conditions are satisfied, then the
mission is (a) considered to be evalutated, and (b) the pass/fail
result is set to pass.

%=================================================================
\subsection{Results of a Mission Evaluation}
\label{sec_pme_eval}

When a mission is considered to be evaluated (lead conditions
satisfied), either or both of the above actions are taken:

\begin{packed_itemize}
\item User configured result flags are posted to the MOOSDB
\item User configures result lines are written to a file. 
\end{packed_itemize}

\vspace{0.1in}
\noindent
Result flags come in one of three flavors: {\em pass} flag, {\em fail}
flags, and simply {\em result} flags. The first is published only when
the pass/fail result passes, the second only when it fails, and the
result flag is published either way. Here are some examples:

\vspace{0.1in}
\noindent
\pvar{pass\_flag = RESULT=success} \\
\pvar{fail\_flag = RESULT=failed} \\
\pvar{result\_flag = MISSION\_EVALUATED=true} \\

\vspace{0.1in}
\noindent
The other action taken on evaluation is the publication of a result
line to a result file. For example, if configured:

\vspace{0.1in}
\noindent
\pvar{result\_file = result.txt} \\
\pvar{result\_col  = result=\$[GRADE]} \\

\vspace{0.1in}
\noindent
The action on mission evaluation will be to write a single line to
the file \var{results.txt}:

\vspace{0.1in}
\begin{fileverb}
 result=pass
\end{fileverb}
\vspace{0.1in}

\vspace{0.1in}
\noindent
Continuing with the above example, we know that if the result passes
then the two conditions involving \mvar{ODOMETRY} and
\mvar{FOUND\_OBJECT} must have passed. But way may want to also know the
actual odometry value. Using another column 

\vspace{0.1in}
\noindent
\pvar{result\_col  = odometry=\$[ODOMETRY]}

\vspace{0.1in}
\noindent
then the line written to the results file would be: 

\vspace{0.1in}
\begin{fileverb}
 result=pass, odometry=1451
\end{fileverb}
\vspace{0.1in}

\vspace{0.1in}
\noindent
Any number of result file columns can be configured. 

%=================================================================
\subsection{Multi-Part Tests}

A multi-part test may be used for evaluating a mission at different
stages. In a single mission, perhaps a vehicle is designed to transit
to a point, then loiter for some period of time, and then return home,
finally setting \mvar{ARRIVED=true}. Rather than just checking to see
if that post was eventually made, the intermediate stages can be
verified too with a {\em multi-aspect logic sequence}. Each aspect
is comprised of one or more lead conditions and one or more pass/fail
conditions. 

\prevblue
\begin{verbatim}
lead_condition = ARRIVED_MID = true
pass_condition = ODOMETRY &lt; 500

lead_condition = LOITER_FINISH = true
pass_condition = ODOMETRY &lt; 1000

lead_condition = ARRIVED = true
pass_condition = ODOMETRY &lt; 1500
\end{verbatim}
\postverb

\noindent
Each pair above is {\em aspect} and the group is called a {\em logic
  sequence}. The overall mission will not be considered evaluated and
results will not be posted or written to a file, until the full
sequence is evaluated. There are no limits to the number of lead, pass
and fail conditions per aspect, and there are no limits on the number of
aspects. The appcasting output of \app{pMissionEval} will show the
current progress of a multi-aspect logic sequence. 

\vspace{0.1in}
\noindent
{\bf Note:} Unlike configuration parameters in nearly all other MOOS
apps, the {\em order} of the configuration lines is significant.

%=================================================================
\subsection{Structuring the Results File}
\label{sec_pme_report}

The results file may have as many columns as the user desires, and
there are no minimums or limits on the columns. Columns are by
default separated by white space, but they can be configured to be
separated by a comma (csp, for comma-separated-pairs) with:

\vspace{0.1in}
\noindent
\pvar{report\_line\_format = csp}   

\vspace{0.1in}
\noindent
The results file will likely contain results from multiple missions,
or mission types. Some consideration should be made to add information
to report lines with an eye toward how the results file will be later
processed, as part of a CI pipeline or a file for generating plotted
data. With this in mind, there are a number of provisions for macros
available for configuring report lines, described next.

%=================================================================
\subsection{Report Macros}
\label{sec_pme_macros}

Any of the flags (result, pass, or fail flags) may include a {\em
  macro}. Likewise, any report column may also contain a macro.
Macros are of the form \var{\$[MACRO]}. A macro may be something built
into the app, such as \var{\$[DATE]}, or others listed below. If a
\var{\$[MACRO]} is not one of the built-in macros, the \app{pMissionEval}
will interpret this to mean that the macro refers to a MOOS variable. 

\vspace{0.1in}
\noindent
For example, consider a mission with an odometry app running and
continuously, posting to the variable \mvar{ODOMETRY}. We may
want to construct an evaluation that uses either (a) a result
flag, or (b) a report column, that uses the odometry information:

\prevblue
\begin{verbatim}
  result_flag = DIST = $[ODOMETRY]
\end{verbatim}
\postverb

\noindent
or:

\prevblue
\begin{verbatim}
  report_column = dist=$[ODOMETRY]
\end{verbatim}
\postverb

\noindent
Built-in macros:

\begin{packed_itemize}
\item \pvar{GRADE}: This is the mission result value. It will be either
  "pending", "pass", or "fail". 

\item \pvar{MHASH\_SHORT}: The short version of a mission hash
  string. An example long version would be
  \var{"mhash=250117-1614B-MEEK-EROS,utc=1737148477.26"}. The short
  version would simply be \var{"MEEK-EROS"}.

\item \pvar{MHASH\_LSHORT}: A longer version of the short version,
  keeping the single letter preceding it. For example,
  \var{"B-MEEK-EROS"}, instead of \var{"MEEK-EROS"}.

\item \pvar{MHASH}: The full version of a mission hash, e.g.,
  \var{"mhash=250117-1614B-MEEK-EROS,utc=1737148477.26"}. 

\item \pvar{MISSION\_HASH}: Same as \pvar{\$[MHASH]}. 

\item \pvar{MHASH\_UTC}: The timestamp component of a mission hash
  string. An example long version would be
  \var{"mhash=250117-1614B-MEEK-EROS,utc=1737148477.26"}. The timestamp
  component would simply be \var{"1737148477.26"}.

\item \pvar{MISSION\_FORM}: This is the value set in the \pvar{mission\_form}
  configuration parameter. It should be set to the colloquial name of the
  mission, e.g., the "Alpha" mission. 

\item \pvar{MISSION\_MOD}: This is the value set in the
  \pvar{mission\_mod} configuration parameter. If a mission has several
  mods, then this is the place to distinguish that. 

\item \pvar{WALL\_TIME}: The wall time is the \mvar{DB\_UPTIME} value, divided
  by the time warp value. 
\end{packed_itemize}

\vspace{0.1in}
\noindent
Built-in Time and Date macros:

\begin{packed_itemize}
\item \var{DATE}: A string in the format of \var{YYYY:MM:DD}.
\item \var{TIME}: A string in the format of \var{HH:MM:SS}.
\item \var{YEAR}: A string in the format of \var{YYYY}.
\item \var{MONTH}: A string in the format of \var{MM}.
\item \var{DAY}: A string in the format of \var{DD}.
\item \var{HOUR}: A string in the format of \var{HH}.
\item \var{MIN}: A string in the format of \var{MM}.
\item \var{SEC}: A string in the format of \var{SS}.
\end{packed_itemize}


%=================================================================
\section{Four Example Missions}
%=================================================================

%=================================================================
\subsection{The Alpha Mission Pass/Fail Verifying Event Flags}

Returning to the Alpha mission Section \ref{sec_pme_alpha1}. The goal
is to verify the end flag, waypoint flag and cycle flag features. First,
the waypoint behavior is configured to increment a counter on each
posting of the flags as fallows. 

\prevbhv
\begin{verbatim}
  wptflag   = WPT_FLAG=$[CTR1]
  cycleflag = CYCLE_FLAG=$[CTR2]
  endflag   = ARRIVED=true
\end{verbatim}
\postverb

\vspace{0.1in}
\noindent
Two tests are made in two logic aspects:

\vspace{0.1in}
\begin{fileverb}
  lead_condition = CYCLE_FLAG = 1
  pass_condition = WPT_FLAG = 5

  lead_condition = ARRIVED = true
  pass_condition = WPT_FLAG = 10
  pass_condition = CYCLE_FLAG = 2

  mission_form  = alpha_ufld
  mission_mod   = mod1
  report_file   = results.txt
  report_column = grade=$[GRADE]
  report_column = form=$[MISSION_FORM]
  report_column = mmod=$[MMOD]
  report_column = mhash_short=$[MHASH_SHORT]
  report_column = lshort_short=$[MHASH_LSHORT]

  result_flag = MISSION_EVALUATED = true
\end{fileverb}
\vspace{0.1in}

%=================================================================
\subsection{The ObAvoid Mission Pass/Fail Verifying Obstacle Avoidance}

Returning to the obstacle avoidance mission described in Section
\ref{sec_pme_obavoid1}. This is misison \var{02-obavoid} in the
\var{missions-auto} repository.  A vehicle is driven through an
obstacle field with a dedicated MOOS app monitoring each obstacle
encounter. This app is called \app{uFldCollObDetect}. It publishes
three key variables:

\begin{packed_itemize}
\item \mvar{OB\_TOTAL\_ENCOUNTERS}: Incremented each time a vehicle passes
  an obstacle. 
\item \mvar{OB\_TOTAL\_COLLISIONS}: Incremented each time a vehicle collides
  with an obstacle. 
\item \mvar{OB\_TOTAL\_NEAR\_MISSES}: Incremented each time a vehicle encounter
  has a near miss with an obstacle.
\end{packed_itemize}

\vspace{0.1in}
\noindent
For each of the above, the app is configured with range threshold, e.g, a 20 meter
CPA to an obstacle counts as an encounter, a 5 meter CPA is a near miss, and 1
meter CPA is considered a collision. 

\vspace{0.1in}
\noindent
A simple pass/fail test can be constructed with the below \app{pMissionEval}
configuration:

\vspace{0.1in}
\begin{fileverb}
  lead_condition = (OB_TOTAL_ENCOUNTERS &gt; $(TEST_ENCOUNTERS))

  pass_flag   = OB_TOTAL_COLLISIONS = 0
  result_flag = MISSION_EVALUATED = true

  report_column = min_sep=$(SEP)
  report_column = grade=$[GRADE]
  report_column = encounters=$[OB_TOTAL_ENCOUNTERS]
  report_file = results.log
\end{fileverb}
\vspace{0.1in}

\noindent
This test can be conducted by running, using time warp 10:

\vspace{0.1in}
\begin{consoleverb}
 $ xlaunch.sh 10
\end{consoleverb}
\vspace{0.1in}

\noindent
This should produce a line a output in file \var{results.log} similar to:

\vspace{0.1in}
\begin{consoleverb}
  min_sep=10  grade=pass  encounters=25
\end{consoleverb}
\vspace{0.1in}

\noindent
Note in this mission, the launch script accepts command line arguments
setting the (a) the minimum separation between obstacles, and (b) the
total number of encounters tested. The defaults are 10 and 25
respectively as shown in the above result line. The values are
expanded in the \cmd{nsplug} macros \var{\$(SEP)} and
\var{\$(TEST\_ENCOUNTERS)}.  The mission could be launched instead
with:

\vspace{0.1in}
\begin{consoleverb}
 $ xlaunch.sh 10 --sep=12 --enc=40
\end{consoleverb}
\vspace{0.1in}

\noindent
This should produce a line a output in file \var{results.log} similar to:

\vspace{0.1in}
\begin{consoleverb}
  min_sep=12  grade=pass  encounters=40
\end{consoleverb}
\vspace{0.1in}


%=================================================================
\subsection{The ObAvoid Mission with Sensitivity Analysis}

Staying with the same obstacle avoidance mission, a sequence of tests
is constructed. The end result will be a \var{results.log} file with
many lines, each representing the results of distict tests, with
perhaps different configuration parameters. For this reason, each line
in the results file is configured with more information:

\vspace{0.1in}
\begin{fileverb}
  lead_condition = (OB_TOTAL_ENCOUNTERS &gt; $(TEST_ENCOUNTERS))

  pass_flag   = OB_TOTAL_COLLISIONS = 0
  result_flag = MISSION_EVALUATED = true

  report_column = form=obavoid
  report_column = min_sep=$(SEP)
  report_column = grade=$[GRADE]
  report_column = collisions=$[OB_TOTAL_COLLISIONS]
  report_column = near_misses=$[OB_TOTAL_NEAR_MISSES]
  report_column = encounters=$[OB_TOTAL_ENCOUNTERS]
  report_file = results.log
\end{fileverb}
\vspace{0.1in}

\noindent
Now a set of tests can be launched, each varying the minimum separation
distance between obstacles in the randomly generated obstacle field:

\vspace{0.1in}
\begin{consoleverb}
  $ xlaunch.sh --sep=14 10
  $ xlaunch.sh --sep=12 10
  $ xlaunch.sh --sep=10 10
  $ xlaunch.sh --sep=8 10
  $ xlaunch.sh --sep=6 10
  $ xlaunch.sh --sep=4 10
\end{consoleverb}
\vspace{0.1in}

\noindent
This should produce several lines of output in the file \var{results.log} similar to:

\vspace{0.1in}
\begin{consoleverb}
  form=obavoid  min_sep=14  grade=pass collisions=0  near_misses=0  encounters=25
  form=obavoid  min_sep=12  grade=pass collisions=0  near_misses=0  encounters=25
  form=obavoid  min_sep=10  grade=pass collisions=0  near_misses=1  encounters=25
  form=obavoid  min_sep=8  grade=pass collisions=0  near_misses=0  encounters=25
  form=obavoid  min_sep=6  grade=pass collisions=0  near_misses=2  encounters=25
  form=obavoid  min_sep=4  grade=pass collisions=0  near_misses=4  encounters=25
\end{consoleverb}
\vspace{0.1in}


%===========================================================
\section{Configuration Parameters for pMissionEval}
%===========================================================
The following parameters are defined for \app{pMissionEval}. A more
detailed description is provided in other parts of this
section. Parameters having default values are indicated.

\vspace{0.15in}
\begin{listing}{Configuration Parameters for \app{pMissionEval}}
\label{lst_params_pmeval}
\renewcommand{\arraystretch}{1.1}
\begin{longtable}{rp{0.76\textwidth}} 

  \pvar{fail\_condition}: &amp; A logic condition that must {\em not} be
  satisfied or otherwise the overall mission will be considered
  successful. Section \ref{sec_pme_cond}.\\

  \pvar{fail\_flag}: &amp; A flag to be publish once the result is known
  and is mission has been deemed to be not successful.  Section
  \ref{sec_pme_eval}. \\

  \pvar{lead\_condition}: &amp; A logic condition that must be satisfied
  before the mission can be evaluated. Section
  \ref{sec_pme_cond}.\\

  \pvar{pass\_condition}: &amp; A logic condition that must be satisfied
  for the overall mission to be considered successful. Section
  \ref{sec_pme_cond}.\\

  \pvar{pass\_flag}: &amp; A flag to be publish once the result is known
  and is mission has been deemed to be successful.  Section
  \ref{sec_pme_eval}. \\

  \pvar{mission\_form}: &amp; Set the name of the mission, e.g., \var{"alpha"},
  or \var{"berta"} etc., so the \var{MISSION\_FORM} macro can be expanded
  in any reporting. \\

  \pvar{report\_column}: &amp; Define a column of output in the in the
  report file. See Section \ref{sec_pme_report}. \\

  \pvar{report\_file}: &amp; Name the report file to receive output of
  results. See Section \ref{sec_pme_report}. \\

  \pvar{report\_line\_format}: &amp; Determines the separator between report
  columns. By default it is white space, \pvar{"white"}, but can also
  be set to us a comma, \pvar{"csp"}. \\

  \pvar{result\_flag}: &amp; A flag to be publish once the result is
  known.  Section \ref{sec_pme_eval}. \\


\end{longtable}
\end{listing}
  

%=================================================================
\section{Publications and Subscriptions of pMissionEval}
%=================================================================
\label{sec_pme_interface}

The interface for \app{pMissionEval}, in terms of publications and
subscriptions, is described below. This same information may also be
obtained from the terminal with:

\vspace{0.1in}
\begin{consoleverb}
 $ pMissionEval --interface or -i
\end{consoleverb}
\vspace{0.05in}

%=================================================================
\subsection{Variables Published by pMissionEval}
\label{sec_pme_pubs}

The publications of \app{pMissionEval} are mostly configurable in the
user specified pass, fail and result flags. There are a few hard-coded
publications:

\begin{packed_itemize}[2]
\item \mvar{APPCAST}: Contains an appcast report identical to the
  terminal output. Appcasts are posted only after an appcast request
  is received from an appcast viewing utility. Section
  \ref{sec_pme_appcast}.

\item \mvar{MISSION\_LCHECK\_STAT}: See Section \ref{sec_pme_lcheck}.

\item \mvar{MISSION\_RESULT}: See Section \ref{sec_pme_lcheck}.

\item \mvar{MISSION\_EVALUATED}: See Section \ref{sec_pme_lcheck}.
  
\end{packed_itemize}

%=================================================================
\subsection{Variables Subscribed for by pMissionEval}
\label{sec_pme_subs}

The subscriptions for \app{pMissionEval} mostly are dictated by the
user-specified test conditions, i.e., the \pvar{lead\_condition},
\pvar{pass\_condition} and \pvar{fail\_condition} parameters. Users
may also specified any MOOS variable to be echoed in any user
specified event flags, and in publications to a results file. There
are a few hard-coded subscriptions:

\begin{packed_itemize}[2]
\item \mvar{APPCAST\_REQ}: A request to generate and post a new
  appcast report, with reporting criteria, and expiration.

\item \mvar{DB\_UPTIME}: The amount of seconds that the local MOOSDB
  has been running. 

\item \mvar{MISSION\_HASH}: The mission hash posting contains a few
  fields that need to be know to support a couple mission hash macros.
  See Section \ref{sec_pme_macros}.

\item \mvar{APPNAME\_PID}: 
  \var{"ignore"} (\var{"none"}). Section \ref{sec_pme_getpid}.

\end{packed_itemize}


%=================================================================
\section{Terminal and AppCast Output}
%=================================================================
\label{sec_pme_appcast}

The \app{pMissionEval} application produces some useful information to
the terminal on every iteration of the application. An example is
shown in Listing \ref{lst_pme_appcast} below. This application is also
appcast enabled, meaning its reports are published to the MOOSDB and
viewable from any \app{uMAC} application or \app{pMarineViewer}. The
counter on the end of line 2 is incremented on each iteration of
\app{pMissionEval}, and serves a bit as a heartbeat indicator.  The
\var{"0/0"} also on line 2 indicates there are no configuration or run
warnings detected.

\pskip

\noindent
The output in the below example comes from the example described in 
Section \ref{sec_pme_example}. 

\vspace{0.1in}
\begin{listing}{Example terminal or appcast output for \app{pMissionEval}}
\label{lst_pme_appcast}
\footnotesize
\begin{verbatim}
   1  ===================================================================
   2  pMissionEval shoreside                                      0/0(25)
   3  ===================================================================
   4  Overall State: pending
   5  ============================================== 
   6   logic_tests_stat: unmet_lead_cond: [ARRIVED=true]
   7       result flags: 1
   8         pass flags: 0
   9         fail flags: 0
  10         curr_index: 0
  11        report_file: results.log
  12  
  13  ==============================================
  14  Supported Macros: 
  15  ==============================================
  16  
  17  Macro         CurrVal
  18  ------------  -------------
  19  ENCOUNTER     0
  20  MHASH_SHORT   GOLD-BABY
  21  MHASH_UTC     1737549326.28
  22  MISSION_FORM  01-colavd
  23  WALL_TIME     0
  24   
  25  ==============================================
  26  Logic Sequence: 
  27  ==============================================
  28  
  29  Index  Status   Cond  PFlag  FFLag
  30  -----  -------  ----  -----  -----
  31  0      current  1     0      0
  32  
  33  ==============================================
  34  Reporting: 
  35  ==============================================
  36    report_file: results.log
  37     report_col: m_form=$[MISSION_FORM]
  38     report_col: 5
  39     report_col: $[ENCOUNTER]
  40     report_col: mhash=$[MHASH_SHORT]
  41     report_col: utc=$[MHASH_UTC]
  42     report_col: wall=$[WALL_TIME]
  43    latest_line: 
  44  
  45  ===================================================================
  46  Most Recent Events (4):
  47  ===================================================================
  48  [4.80]: Mail:DB_UPTIME
  49  [2.78]: Mail:DB_UPTIME
  50  [2.27]: Mail:MISSION_HASH
  51  [0.76]: Mail:DB_UPTIME
\end{verbatim}
\end{listing}
\normalsize
\vspace{0.1in}

\noindent
The first few lines (4-11) provide a high-level snapshot of the current
state. Line 4 shows the overall state of the test. Lines 13-23 show all
macros available in posting flags or generating column output in a report
file. Lines 25-31 show the prevailing logic sequence. A sequence may have
several aspects (one per line in this table). Lines 33-43 show the configuration
of the report file output. Line 43 shows the latest line written to this
file. Line 45 is the start of the most recent events. For \app{pMissionEval},
for now, the events are just the received mail registered for this app.






\vspace{0.1in}
\noindent
All flags for all behaviors also have certain macros defined for event
postings. Two such macros are \var{\$[CTR1]} and \var{\$[CTR2]}. They
are counters that are incremented each time they are used in a posting.
So if the \bvar{cyle\_flag} and \bvar{wpt\_flag} parameters are set in
the waypoint behavior of the Alpha mission:

\vspace{0.1in}
\noindent
\bvar{  wpt\_flag   = WFLAG=\$[CTR1]} \\
\bvar{  cycle\_flag = CFLAG=\$[CTR1]} \\
\bvar{  repeat      = 1} \\
\bvar{  end\_flag   = ARRIVED=true} \\

\vspace{0.1in}
\noindent
The vehicle will traverse the five points twice, and the MOOS variables
\mvar{WFLAG} and \mvar{CFLAG} should have the values 10 and 2 respectively,
and \mvar{ARRIVED} should be set to \var{true}.




\begin{packed_itemize}
\item It encapsulates mission evaluation to be defined by the
  \app{pMissionEval} configuration block. This allows automated
  scripts to treat a mission generically: launch it, monitor for its
  ending, and shut it down.
\item It can facilitate multiple component evaluations in a single
  mission, distilling it down to a single pass/fail grade.
\item It allows for custom configuration of a report file that facilitates
  easier post-processing for plotting result data. 
\item
\end{packed_itemize}



%-----------------------------------------------------------------
\subsection{Relation to uMayFinish}

In an automated mission, some entity is monitoring one or more of the
MOOS communities involved in the situation, typically the shoreside,
for an event that indicates that the mission is complete and
evaluations are completed and registered somewhere. After this point,
the mission (all MOOS apps) are automatically shut down.



The \app{uMayFinish} application ma








\end{document}

</pre></body></html>