<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{uProcessWatch: Monitoring MOOS Application Health}
\vspace{0.1in}
\labdate{June 2018}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labtoc{small}
\end{labcover}

\wikidoc{file=IvPTools.UProcessWatch}
\wikidoc{bugz=uProcessWatch}
\wikidoc{auth=mikerb@mit.edu}

%===================================================================
\section{Overview}
%===================================================================
\label{sec_upw}
%\index{uProcessWatch}

%\index[vars]{\mvar{PROC\_WATCH\_SUMMARY}}
%\index[vars]{\mvar{PROC\_WATCH\_EVENT}}

The uProcessWatch application monitors the health of a set of MOOS
application. It does two things:

\begin{packed_itemize}[2]
\item It monitors the presence of a set of MOOS apps,
\item It monitors the CPU load of a set of MOOS apps.
\end{packed_itemize}

\noindent 
In the case of the former, \app{uProcessWatch} continually monitors
the \mvar{DB\_CLIENTS} list for applications it has responsibility for
watching.  In the case of the latter, MOOS apps are already monitoring
and reporting their own CPU load and \app{uProcessWatch} is doing
nothing more than gathering that information for display in the
terminal or appcast message. An example terminal output is shown
below:

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.7\textwidth]{figures/upw_appcast.png}
  \caption{{\bf A typical terminal or appcast report for \vvar{uProcessWatch}.}}
  \label{fig_upw_appcast}
\end{figure}

\vspace{0.1in}

\noindent
The first line, "Summary: All Present", tells you that no processes
are missing. This is also posted in the variable \mvar{PROC\_WATCH\_SUMMARY}.
The list of applications launched with \app{pAntler} is shown in the 
next block. The main body shows, for each watched application, the reason
why the process is on the watch list, the status, current CPU load as reported
by the process itself, and the maximum CPU load noted so far. 

\pskip

The bottom section of the terminal output shows the events (similar to
all appcasting output). In the case of \app{uProcessWatch}, events
note either the arrival or disappearance of a watched process.  Each
event also results in the posting of the variable
\mvar{PROC\_WATCH\_EVENT}. The user may configure \app{uProcessWatch}
to {\em not} watch certain named applications or patterns of
applications, such as \var{uXMS*}, to avoid unwarranted alerts.

%===================================================================
\section{Typical uProcessWatch Usage Scenarios}
%===================================================================

\subsection{Using uProcessWatch with AppCasting and pMarineViewer}

The first usage scenario is perhaps the most typical since it is
viable both in simulation and in the field where the vehicles have a
network connection to a shoreside MOOS community. The idea is shown in
Figure \ref{fig_upw_topology} below. Multiple vehicles each run
\app{uProcessWatch} locally. A network connection from each vehicle
to a shoreside MOOS community is used to share appcast messages using
\app{pShare}. The shoreside community is running \app{pMarineViewer}
which supports a multi-vehicle appcast viewing mode.  In this way, the
shoreside user may monitor the health of all vehicles' applications
with one tool. If a process on a single vehicle goes missing, the menu
item on the shoreside viewer for that vehicle turns red.

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.65\textwidth]{figures/upw_topology.png}
  \caption{{\bf Using uProcessWatch with AppCasting:} uProcessWatch is
    run locally on each fielded vehicle, generating appcasts posted to
    the local MOOSDB. Appcasts are shared to the shoreside and
    collected by the pMarineViewer tool for monitoring processes
    across all vehicle.  This scenario is relevant when there is a
    network connection from the vehicles to the shoreside.}
\label{fig_upw_topology}
\end{figure}

%===================================================================
\subsection{Directly Accessing the \mvar{PROC\_WATCH\_SUMMARY} Output}

The main health indicator produced by \app{uProcessWatch} in the MOOS
variable \mvar{PROC\_WATCH\_SUMMRY}. This can be checked on a remote
machine by simply scoping on this variable. The \app{uMS} application
distributed with MOOS will do the trick for example. To focus solely on 
this variable, the \app{uXMS} tool may also be used as follows:

\begin{consoleverb}
 $ uXMS --server_host=10.25.0.72 --server_port=9000 PROC_WATCH_SUMMARY
\end{consoleverb}

\noindent
Or one may ssh onto the vehicle and launch a scope locally on this variable.


%===================================================================
\section{Using and Configuring the uProcessWatch Utility}
%===================================================================
\label{sec_upw_usage}

The primary configuration of \app{uProcessWatch} is defining the {\em
  watch list}, the set of other MOOS processes to monitor. By default
all processes ever noted to be connected to the MOOSDB are on the
watch list. The same with all processes name in the Antler
configuration block of the mission file.  This default is used because
it is simple, and a good rule of thumb is that if any process
disconnects from the MOOSDB, it's probably a sign of trouble.

%===================================================================
\subsection{The \mvar{DB\_CLIENTS} Variable for Detecting Missing Processes}
\label{sec_upw_dbclients}

The MOOSDB, about once per second, posts the variable
\mvar{DB\_CLIENTS} containing a comma-separated list of clients (other
MOOS apps) currently connected to the MOOSDB. When \app{uProcessWatch}
is configured to watch for all processes, it simply augments the watch
list for any process that ever appears on the incoming
\mvar{DB\_CLIENTS} mail. If the process is then missing at a later
reading of this \mvar{DB\_CLIENTS} mail, is is considered AWOL.

%===================================================================
\subsection{Defining the Watch List}
\label{sec_upw_watchlist}

The {\em watch list} is a list of processes, i.e., MOOS apps. Each
item on the list will be reported missing if it does not appear in the
list of clients shown by the current value of \mvar{DB\_CLIENTS}. By
default, all clients that ever appear on the \mvar{DB\_CLIENTS} list
will be added to the watch list. The same is true for all processes
named in the Antler configuration block of the mission file.  This
default can be overridden by the following configuration option:

\vspace{0.1in}
  \pvar{watch\_all}\psvarr{ = false}
\vspace{0.1in}

\noindent
The value of \pvar{watch\_all} may also be set to \var{"antler"} to
indicate that items on the Antler list are to be watched by default,
but not those that appear in the \mvar{DB\_CLIENTS} list. Likewise it
may be set to \var{"dbclients"} to indicate that items in the
\mvar{DB\_CLIENTS} list are to be watched by default, but not those in
the Antler list.  Processes may be added to the watch list by
explicitly naming them in configuration block as follows:

\vspace{0.1in}
  \pvar{watch}\psvarr{ = process\_name[*]}
\vspace{0.1in}

\noindent
A process may be named explicitly, or the prefix of the process may be
given, e.g., \pvar{watch}\psvar{=uXMS*}.  This will match all processes fitting
this pattern, e.g., \var{uXMS\_845}, \var{uXMS\_23}. 

\pskip

Processes may be explicitly {\em excluded} from the watch list with
configurations of the following:

\vspace{0.1in}
  \pvar{nowatch}\psvarr{ = process\_name[*]}
\vspace{0.1in}


\noindent
This is perhaps most appropriate when coupled with
\pvar{watch\_all}\psvar{=true}.  If a process is for some reason both
explicitly included and excluded, the inclusion takes precedent.

%===================================================================
\subsection{Reports Generated}

There are three reports generated in the variables:

\begin{packed_itemize}
\item \mvar{PROC\_WATCH\_EVENT}
\item \mvar{PROC\_WATCH\_SUMMARY},
\item \mvar{PROC\_WATCH\_FULL\_SUMMARY}
\end{packed_itemize}

All reports are generated only when
there is a change of status in one of the watched processes.  The
first type of report is generated for each event when a watched
process is noted to have connected or disconnected to the MOOSDB. The
following are examples:

\prevgreen
\begin{verbatim}
  PROC_WATCH_EVENT = "Process [pMarinePID] is noted to be present."
  PROC_WATCH_EVENT = "Process [pMarinePID] has died!!!!"} 
  PROC_WATCH_EVENT = "Process [pMarinePID] is resurected!!!"
\end{verbatim}
\postverb

In the first line above, a process is reported to be present that was
never previously on the watch list. In the third line a process that
was previously noted to have left the watch list is reported to have
returned or been restarted.

\pskip

The \mvar{PROC\_WATCH\_SUMMARY} variable will list the set of processes
missing from the watch list or report \var{"All Present"} if no items
are missing. For example: 

\prevgreen
\begin{verbatim}
  PROC_WATCH_SUMMARY = "All Present"
  PROC_WATCH_SUMMARY = "AWOL: pMarinePID,uSimMarine"
\end{verbatim}
\postverb

\pskip

The \mvar{PROC\_WATCH\_FULL\_SUMMARY} variable will list a more complete
and historical status for all processes on the watch list. For example: 

\prevgreen
\begin{verbatim}
  PROC_WATCH_FULL_SUMMARY = "pHelmIvP(1/0),uSimMarine(1/1),pMarinePID(2/2)"
\end{verbatim}
\postverb

\noindent
The numbers in parentheses indicate how many times the process has been noted
to connect to the MOOSDB over the number of times it has been noted to have
disconnected. A report of \var{"(1/0)"} is the healthiest of possible reports, 
meaning it has connected once and has never disconnected. 

%===================================================================
\subsection{Watching and Reporting on a Single MOOS Process}

If desired, \app{uProcessWatch} may be configured to generate a report
dedicated a single MOOS process with the following example configuration:

\prevblue
\begin{verbatim}
  watch = pBasicContactMgr : BCM_OK
\end{verbatim}
\postverb

\noindent
In this case a MOOS variable, \mvar{BCM\_OK}, will be set to either
true or false depending on whether the process \app{pBasicContactMgr}
presently appears on the list of connected clients in listed in
\mvar{DB\_CLIENTS}.

%===================================================================
\subsection{A Heartbeat for the Watch Dog}

By default \app{uProcessWatch} will only post
\mvar{PROC\_WATCH\_SUMMARY} when its value changes. A long stale
\mvar{PROC\_WATCH\_SUMMARY = "All Present"} likely means that
everything is fine. It could also mean the \app{uProcWatchSummary}
process itself died without ever posting anything to suggest
otherwise. The user has the configuration option to post
\mvar{PROC\_WATCH\_SUMMARY} every $N$ seconds regardless of whether or
not the value has changed:

\prevblue
\begin{verbatim}
    summary_wait = 30  // Summary posted at least every 30 seconds 
\end{verbatim}
\postverb

\noindent
This in effect creates a heartbeat for monitoring \app{uProcessWatch}.
The default value for this parameter is $-1$. Any negative value will
be interpreted as a request for postings to be made only when the
posting value changes. Regardless of the \pvar{summary\_wait} setting,
the other two reports, \mvar{PROC\_WATCH\_EVENT} and
\mvar{PROC\_WATCH\_FULL\_SUMMARY}, will only be made when the values
change.

%===================================================================
\subsection{Excusing a Process}
\label{sec_upw_excused}

An application on the watch list may be excused and disconnect from
the MOOSDB if it posts to the variable \mvar{EXITED\_NORMALLY} with
the name of itself. A check is made by \app{uProcessWatch} of the {\em
  source} of the posting to ensure that message was posted by the
exiting process.  It's up to the developer of an application to build
in the feature declaring a normal exit. An example is the
\app{uTimerScript} application which has the ability to execute a
script of postings to the MOOSDB followed by an exit. In this case,
the {\em status} of application becomes \var{EXCUSED}, as shown in
Figure \ref{fig_upw_excused}.

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.6\textwidth]{figures/upw_excused.png}
  \caption{The process \app{uTimerScript} has gone missing due to its
    own intentional exit. Before exiting it posted
    \msvvar{EXITED\_NORMALLY=}\msvvar{uTimerScript}, and therefore
    \app{uTimerScript} is regarded as excused.}
\label{fig_upw_excused}
\end{figure}
\vspace{0.05in}

\noindent
Excusing a missing process is different than choosing to not include
it on the watch list.  Some applications are, by their nature,
designed to disappear at some point.  Scoping tools like \app{uXMS},
\app{uPokeDB} or \app{uHelmScope} are examples applications that
regularly appear and disappear. They are best excluded entirely from
the watch list with the \pvar{nowatch} configuration parameter.

%===================================================================
\subsection{Allowing Retractions if a Process Reappears}

With the addition of appcasting to \app{uProcessWatch}, a missing process
also triggers an appcast run warning, as shown on the top in Figure
\ref{fig_upw_appcast_missing} below. 

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.6\textwidth]{figures/upw_appcast_missing.png}
  \caption{The process pNodeReporter has gone missing. This is noted
    as a runtime warning at the top, and the MISSING status in the
    body of the report.}
\label{fig_upw_appcast_missing}
\end{figure}
\vspace{0.05in}

\noindent
However, there are cases where a process goes missing but then returns, in which
case the warning is a distraction. These reasons include.

\begin{packed_itemize}
\item The application may actually exit and then restart a short time later.
\item The application my be running at a very slow apptick in which case it 
 may be dropped from the \mvar{DB\_CLIENTS} list momentarily.
\item The application may be missing only because it hasn't launched yet.
\end{packed_itemize}

\noindent
To address this, \app{uProcessWatch} will allow retractions on appcast
run warnings. If the application disappears and reappears, the appcast
run warning will disappear. The successive posted values of
\mvar{PROC\_WATCH\_SUMMARY} will show that the process was at some
point declared AWOL, but once the process returns, the appcast output will 
no longer raise attention to the issue. 

\pskip

Of course there are situations where a process that disappears and
then reappears is an indication of a problem, not to be swept under
the rug. In this case the default behavior of \app{uProcessWatch} may
be changed by setting \pvar{allow\_retractions} to
false. Presently this setting cannot be set on a per-process
manner.

%===================================================================
\section{Configuration Parameters of uProcessWatch}
%===================================================================
%\index{uProcessWatch!Configuration Parameters}
%\index{Configuration Parameters!uProcessWatch}

The following parameters are defined for \app{uProcessWatch}. A more
detailed description is provided in other parts of this
section. Parameters having default values indicate so in parentheses
below.

\vspace{0.15in}
\begin{listing}{Configuration Parameters for uProcessWatch}
\label{lst_upw_params}
\renewcommand{\arraystretch}{1.1}
\begin{longtable}{rp{0.75\textwidth}} 

  \pvar{allow\_retractions}: &amp; If true, run warnings are retracted if 
  a process reappears after disappearing. Legal values: true, false.
  The default is true. \\
  
  \pvar{nowatch}: &amp; A process or list of MOOS processes to {\em not}
  watch. \\
  
  \pvar{post\_mapping}: &amp; A mapping from one posting variable name
  to another. \\
  
  \pvar{summary\_wait}: &amp; A maximum amount of time between
  \mvar{PROC\_WATCH\_SUMMARY} postings. Negative value indicates
  posting occurs only when the value changes regardless of elapsed
  time.  Legal values: any numerical value. The default is $-1$. \\
  
  \pvar{watch}: &amp; One or more comma-separated MOOS process to watch
  and report on. \\
  
  \pvar{watch\_all}: &amp; If true, watch all processes that become known
  either via the \mvar{DB\_CLIENTS} list or the Antler list. Legal
  values: true, false. The default is true. \\
  
\end{longtable}
\end{listing}

%===================================================================
\subsection{An Example MOOS Configuration Block}

Listing \ref{lst_upw_econfig} shows an example MOOS configuration block produced
from the following command line invocation:

\begin{consoleverb}
 $ uProcessWatch --example or -e
\end{consoleverb}

\vspace{0.25in}
\begin{listing}{Example configuration of the uProcessWatch application}
\label{lst_upw_econfig}
\footnotesize
\begin{verbatim}
  1  ProcessConfig = uProcessWatch                                   
  2  {                                                               
  3    AppTick   = 4                                                 
  4    CommsTick = 4                                                 
  5                                                                  
  6    watch_all = true   // The default is true.                    
  7                                                                  
  8    watch  = pMarinePID:PID_OK                                    
  9    watch  = uSimMarine:USM_OK                                    
 10                                                                  
 11    nowatch = uXMS*                                               
 12                                                                  
 13    allow_retractions = true   // Always allow run-warnings to be 
 14                               // retracted if proc re-appears    
 15                                                                  
 16    // A negative value means summary only when status changes.   
 17    summary_wait = 10 // Seconds. Default is -1.                  
 18                                                                  
 19    post_mapping = PROC_WATCH_FULL_SUMMARY, UPW_FULL_SUMMARY      
 20  }                          
\end{verbatim}
\end{listing}
\normalsize
 
%===================================================================
\section{Publications and Subscriptions for uProcessWatch}
%\index{uProcessWatch!Publications and Subscriptions}
%\index{Publications and Subscriptions!uProcessWatch}
\label{sec_upw_interface}

The interface for \app{uProcessWatch}, in terms of publications and
subscriptions, is described below. This same information may also be
obtained from the terminal with:

\begin{consoleverb}
 $ uProcessWatch --interface or -i
\end{consoleverb}


%===================================================================
\subsection{Variables Published by uProcessWatch}
\label{sec_upw_pubs}

The primary output of \app{uProcessWatch} to the MOOSDB is a summary
indicating whether or not certain other processes (MOOS apps) are
presently connected.

\begin{packed_itemize}[2]
\item \mvar{APPCAST}: Contains an appcast report identical to the
  terminal output. Appcasts are posted only after an appcast request
  is received from an appcast viewing utility.

\item \mvar{PROC\_WATCH\_EVENT}: A report indicating a particular
  process has been noted to be gone missing or noted to have
  (re)joined the list of active processes. 

\item \mvar{PROC\_WATCH\_FULL\_SUMMARY}: A single string report for
  each process indicating how many times it has connected and
  disconnected from the MOOSDB. 

\item \mvar{PROC\_WATCH\_SUMMARY}: A report listing all missing
  processes, or "All Present" if no processes are missing. 
\end{packed_itemize}

The user may also configure \app{uProcessWatch} to make a posting
dedicated to a particular watched process. For example, with the
configuration \pvar{watch}\psvar{=pNodeReporter:PNR\_OK}, the status of this
process is conveyed in the MOOS variable \mvar{PNR\_OK}, set to either
true or false depending on whether or not it is
present.

\pskip

The variable name for any posted variable may be changed to a
different name with the \pvar{post\_mapping} configuration
parameter. For example, \pvar{post\_mapping}\psvar{=PROC\_WATCH\_EVENT,
  UPW\_EVENT} will result in events being posted under the
\msvar{UPW\_EVENT} variable rather than \mvar{PROC\_WATCH\_EVENT}
variable.

%===================================================================
\subsection{MOOS Variables Subscribed for by uProcessWatch}
\label{sec_upw_subs}

The following variable(s) will be subscribed for by \app{uProcessWatch}:

\begin{packed_itemize}[2]
\item \mvar{APPCAST\_REQ}: A request to generate and post a new
  apppcast report, with reporting criteria, and expiration.

\item \mvar{DB\_CLIENTS}: A comma-separated list of clients currently
  connected to the MOOSDB, posted by the MOOSDB. Used for detecting
  missing processes. Section \ref{sec_upw_dbclients}.

\item \mvar{EXITED\_NORMALLY}: An indication made by a process,
  potentially on the watch list, that it has exited normally and
  should be excused, and not regarded as missing. Section
  \ref{sec_upw_excused}.

\item \msvar{&lt;PROCNAME&gt;\_STATUS}: The status string generated for all
  MOOSApps, containing the current CPU load among other things.
\end{packed_itemize}


\end{document}

</pre></body></html>