<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">%====================================================================
\section{Mission S11: The Kilo Mission}
%====================================================================
\wikisec{grp=xmiss}
\wikisec{file=HelmXMissKilo,short=Example Kilo}
\label{exmissions_s11}

The purpose of the Kilo mission is to illustrate the use of the
standby helm, and the use of the TestFailure behavior.  The standby
helm consists of an otherwise normally configured helm with the
additional standby parameter invoked.  This helm will wait in standby
mode until the heartbeat of another (primary) helm ceases to be posted
to the \app{MOOSDB} for some period of time.  In this example mission,
a primary and standby helm are configured with the primary helm
executing a simple mission similar to the Alpha mission, but also
using an instance of the TestFailure behavior. This behavior will be
configured to trigger either a crash or hang of the primary helm to
demonstrate the manner in which a standby helm will step in to take
over with its own mission. The standby mission in this case is simply
a behavior to return the vehicle to its launch position.

%------------------------------------------------------------------
\subsection{Overview of the Kilo Mission Components and Topics}

\small
\begin{longtable}{|r|p{0.76\textwidth}|} \hline 
  \bsvar{Behaviors}: &amp; Waypoint, TestFailure \\ \hline
  
  \bsvar{MOOS Apps:} &amp; \app{pHelmIvP}, \app{pLogger}, \app{uSimMarine},
  \app{pMarinePID}, \app{pNodeReporter}, \app{pMarineViewer} \\ \hline
  
  \bsvar{Primary Topics:} &amp;
  (1) The use of a standby helm. \newline
  (2) The TestFailure behavior. \newline
  (3) Scoping the primary and standby helm states at runtime. \\ \hline
  
  \bsvar{Side Topics:} &amp; (1) \app{uHelmScope} may be used to monitor
  the relationship between the shadow helm and primary helm. \\ \hline
  
\end{longtable}
\normalsize

%======================================================================
\subsection{Launching the Kilo Mission}

The Kilo mission may be launched from the command line:

\vspace{0.15in}
\noindent
\begin{consoleverb}
 $ cd moos-ivp/missions/s11_kilo/
 $ ./launch.sh --warp=10           (or more simply, ./launch.sh 10)
\end{consoleverb}
\vspace{0.12in}

\noindent
This should bring up a \app{pMarineViewer} window like that shown in
Figure \ref{fig_kilo_one}, with a single vehicle, \var{"kilo"},
initially in the \var{"PARK"} mode. After hitting the \var{DEPLOY}
button in the lower right corner, the vehicle enters the
\var{"SURVEYING"} mode and begins to proceed along the waypoints as
shown.

%=====================================================================
\subsection{Topic \#1: The Use of a Standby Helm}

The vehicle in this example is configured with both a primary helm and
a secondary helm. Configuring a mission with two helm instances is
straight forward.  The primary helm is configured as done normally, as
in lines 3-11 in Listing \ref{lst_kilo_helms} below. The standby helm is
configure identically but with one additional line, as in line 23 in
Listing \ref{lst_kilo_helms} below.  This line denotes the helm it to be
a {\em standby} helm, and indicates the time threshold used in
determining when to take over from a primary helm.

\vspace{0.25in}
\begin{listing}{Configuration of the primary and secondary helm in the Kilo mission}
\label{lst_kilo_helms}
\footnotesize
\begin{verbatim}
  1  //----------------------------- pHelmIvP config block
  2  
  3  ProcessConfig = pHelmIvP
  4  {
  5    AppTick    = 4
  6    CommsTick  = 4
  7  
  8    behaviors  = kilo.bhv
  9    domain     = course:0:359:360
 10    domain     = speed:0:4:21
 11  }
 12  
 13  //----------------------------- pHelmIvP_Standby config block
 14  
 15  ProcessConfig = pHelmIvP_Standby
 16  {
 17    AppTick    = 4
 18    CommsTick  = 4
 19  
 20    behaviors  = kilo_standby.bhv
 21    domain     = course:0:359:360
 22    domain     = speed:0:4:21
 23    standby    = 2
 24  }
\end{verbatim}
\end{listing}
\vspace{0.1in}

\noindent
Both helms are instances of the \app{pHelmIvP} application, but one of
them is named \app{pHelmIvP\_Standby}. The name of the application is
not in any way used to put this helm instance into a standby role. It
is chosen simply to be different from the primary helm, since the
\app{MOOSDB} requires all connected applications to have unique names, and
to be clear when or if debugging the mission.

\vspace{0.1in}
\noindent
ched by \app{pAntler} with the name
\app{pHelmIvP\_Standby}.  This is done by giving it the alternative
name using the \app{pAntler} \pvar{ExtraProcessParams} parameter as
shown on lines 15 and 18 below.

\vspace{0.25in}
\begin{listing}{Configuration of Antler}
\label{lst_kilo_antler}
\footnotesize
\begin{verbatim}
  1  //------------------------------------------
  2  // Antler configuration  block
  3  
  4  ProcessConfig = ANTLER
  5  {
  6    MSBetweenLaunches = 200
  7  
  8    Run = MOOSDB            @ NewConsole = false
  9    Run = pLogger           @ NewConsole = false
 10    Run = uSimMarine        @ NewConsole = false
 11    Run = pNodeReporter     @ NewConsole = false
 12    Run = pMarinePID        @ NewConsole = false
 13    Run = pMarineViewer     @ NewConsole = false
 14    Run = pHelmIvP          @ NewConsole = true
 15    Run = pHelmIvP          @ NewConsole = true, ExtraProcessParams=HParams
 16    Run = uProcessWatch     @ NewConsole = false
 17  
 18    HParams=--alias=pHelmIvP_Standby
 19  }
\end{verbatim}
\end{listing}
\vspace{0.1in}

\noindent
If the shadow helm is launched independently, not with \app{pAntler},
it may be launched with:

\vspace{0.1in}
\small
\begin{consoleverb}
 $ pHelmIvP kilo.moos --alias=pHelmIvP_Standby
\end{consoleverb}
\normalsize

%=======================================================================
\subsection{Topic \#2: The TestFailure Behavior}

The Kilo mission uses the \var{TestFailure} behavior to artificially
generate a failure of the primary helm. In the Kilo mission it is
configured to produce a "hung" helm with the following configuration:

\vspace{0.25in}
\begin{listing}{Configuration of the TestFailure Behavior}
\label{lst_kilo_bhv}
\footnotesize
\begin{verbatim}
  1  //----------------------------------------------
  2  Behavior = BHV_TestFailure
  3  {
  4    name                = test_failure
  5    condition           = DEPLOY=true
  6    duration            = 120
  7    duration_idle_decay = false
  8  
  9          failure_type  = hang,3
 10  }
\end{verbatim}
\end{listing}
\vspace{0.1in}

\noindent
The first four configuration parameters are defined for all IvP
behaviors. A run condition of \bvar{DEPLOY=true} keeps this behavior
idle until the mission is launched. Once it is launched, a countdown
of 120 seconds begins until the behavior will hang. The
\bvar{duration\_idle\_decay=false} setting ensures that the countdown
doesn't begin until the behavior is in the running state. In this case
the behavior will fail by hanging for three seconds. This will cause
the primary helm to also appear to hang for three seconds, noted by a
three second gap between heartbeats, defined by posting to the MOOS
variable \mvar{IVPHELM\_STATE}. Since the standby helm is configured
to wait no longer than two seconds (see Listing \ref{lst_kilo_helms}),
the standby helm will promptly take over control from the primary
helm. The sequence of events in taking over control is described in
more detail next.

%--------------------------------------------------------------------
\subsection{Topic \#3: Scoping the Helm State(s) at Runtime}

The relationship between the primary and standby helm may be monitored
by scoping on the variable \mvar{IVPHELM\_STATE} during the course of
the mission. Figure \ref{fig_kilo_uxms_one} below shows the situation
shortly after the vehicle is deployed using the \app{uXMS} scoping
tool.

\begin{figure}[H]
  \centering 
  \includegraphics[width=0.8\textwidth]{figures/kilo_uxms_one.png}
  \caption{{\bf Helm State in the Kilo Mission:} Both the standby and
    primary helms are operating normally, posting a helm state message
    about four times per second.}
\label{fig_kilo_uxms_one}
\end{figure}

Initially the standby helm is showing a helm state of \var{"STANDBY"},
and the primary helm is showing a helm state of \var{"DRIVE"}. Since
both helms are operating at the same frequency, they mostly alternate
between postings as shown above.

\pskip

The events shown in Figure \ref{fig_kilo_uxms_two} show that during
the 9 identical postings ending at time $329.86$, the standby helm is
the only helm emitting a heartbeat. During this period the primary
helm has either crashed or hung. Finally the secondary helm takes over
and posts a helm state of \var{"DRIVE+"} for 107 consecutive heartbeats
(iterations). Recall the \var{"+"} is to further distinquish that this
helm was originally configured as a standby helm.

\begin{figure}[H]
  \centering 
  \includegraphics[width=0.8\textwidth]{figures/kilo_uxms_two.png}
  \caption{{\bf Helm State in Kilo Mission after Standby Helm Takes
      Over Hung Primary Helm:} The standby helm has detected a delay
    in the primary helm heartbeat and takes over about two seconds
    later. Note the number in parentheses is the number of identical
    postings with the timestamp showing the time of the last of the
    identical postings.}
\label{fig_kilo_uxms_two}
\end{figure}

\noindent
Eventually it turns out that the original primary helm did not crash
after all but was just temporarily hung. After emerging from its hung
state, upon reading mail in the next loop, it realizes the standby
helm has taken over and immediately concedes control and posts the
\var{"DISABLED"} helm state, and will never again post. One may wonder
why the primary helm, when it finally woke up, did not first mistakely
post \var{"DRIVE"} before conceding control - after all it needs to
first read its mail to learn that another helm is in control. The
primary helm in fact did post \var{"DRIVE"} and \var{"DISABLED"} on
two consecutive iterations. The first posting occurred (time 327.81)
prior to querying the behaviors for input, and the second posting
(time 356.70) occurred on the next iteration of upon reading mail.

\vspace{0.1in}
\noindent
In the case where the standby helm takes over for a crashed helm, the
sequence of events, viewed from the perspective of posts to
\mvar{IVPHELM\_STATE}, is similar as shown in Figure
\ref{fig_kilo_uxms_three}. When things are going fine the standby helm
and primary helm alternately post helm states of \var{STANDBY} and
\var{DRIVE} respectively as the first several posts below shoe. At
some point the primary helm crashes and the only posts are made by the
standby helm. In the example here, there are 10
\msvar{IVPHELM\_STATE="STANDBY"} posts made while the standby helm is
getting closer to triggering a take-over. Finally, upon takeover, it
posts a helm state of \var{"DRIVE+"} thereafter. The primary helm is 
never heard from again.

\begin{figure}[H]
  \centering 
  \includegraphics[width=0.7\textwidth]{figures/kilo_uxms_three.png}
  \caption{{\bf Helm State in Kilo Mission after Standby Helm Takes
      Over Crashed Primary Helm:} The standby helm has detected a
    delay in the primary helm heartbeat and takes over about two
    seconds later. Note the number in parentheses is the number of
    identical postings with the timestamp showing the time of the last
    of the identical postings.}
\label{fig_kilo_uxms_three}
\end{figure}

\subsection{Suggestions for Tinkering}

\begin{itemize}

\item {\em Analyze a helm takeover from the log files.} Run the Kilo
  mission again and, as opposed to monitoring things with a live scope
  as in Figures \ref{fig_kilo_uxms_one} - \ref{fig_kilo_uxms_three},
  let the mission play out. Change directories to the folder where the
  \var{.alog} file was created. Take a look at the
  \mvar{IVPHELM\_STATE} postings using the \app{aloggrep} tool. Confirm
  that the sequence of events described in Figures
  \ref{fig_kilo_uxms_one} - \ref{fig_kilo_uxms_three} are consistent
  with entries of the log file. Hint: This can be achieved without
  mission configuration or code modifacations.

\item {\em Crashing with helm with a mouse click.} Rather than waiting
  for the \var{TestFailure} behavior to run down its clock to failure,
  configure it to fail immediately upon entering the run state
  (\bvar{duration=0}). But also set an additional run-condition, e.g.,
  \bvar{FAIL=true} and configure the \app{pMarineViewer} to make this
  posting upon a mouse-click. Test the modified Kilo mission by
  launching it, and clicking the mouse when ready to see the failure.
  This can be achieved solely through mission configuration
  modification, without code modification.

\item {\em Configure a standby helm that finishes a task.}  Create a
  primary and shadow helm where the primary helm is surveying a set of
  waypoints. The shadow helm should be able to survey the remaining
  points when/if the primary helm is taken over. Hint: this will likely
  require code generation or modification. Extra bonus if achievable 
  solely through mission modification.

%Can we develop a simple method for the helm to tell the behavior
%how long it just took to generate its output?

\end{itemize}


\newpage
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.935\textwidth]{figures/kilo_one.png}
  \caption{{\bf The Kilo Mission (1):} The vehicle \var{"kilo"}
    traverses the waypoints using a primary helm, launched alongside a
    standby helm. The TestFailure behavior is a ticking bomb that will
    hang the primary helm momentarily.}
\label{fig_kilo_one}
\end{figure}

\begin{figure}[H]
  \centering 
  \includegraphics[width=0.935\textwidth]{figures/kilo_two.png}
  \caption{{\bf The Kilo Mission (2):} The TestFailure behavior has
    hung the primary helm. The standby helm has detected the absence
    of a heartbeat (the \mvvar{IVPHELM\_STATE} variable), and take
    over with a simple return mission.}
\label{fig_kilo_two}
\end{figure}
</pre></body></html>