<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{uLoadWatch: Monitoring Application System Load}
\vspace{0.1in}
\labdate{June 2018}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labsrc{project-pavlab/appdocs/app\_uloadwatch} 
\labtoc{small}
\end{labcover}

\wikidoc{file=IvPTools.ULoadWatch}
\wikidoc{bugz=uLoadWatch}
\wikidoc{auth=mikerb@mit.edu}

%====================================================================
\section{Overview}
%====================================================================
\label{sec_uLoadWatch}
%\index{uLoadWatch}

The \app{uLoadWatch} application monitors data produced by individual
applications and reports a warning when the MOOS community begins to
exhibit a strain in the CPU load. Load data is derived solely from two
MOOS variables published by participating MOOS apps. For an example
application \app{pFooBar}, the following two variables are published:

\vspace{0.1in}
\begin{packed_itemize}[1]

\item \mvar{PFOOBAR\_ITER\_GAP}: indicates the ratio of (a) observed
  time between invocations to the application's \var{Iterate()}
  method, and (b) the scheduled time between iterations based on the
  configured \psvar{AppTick}. For example, for an application running
  at 4 Hz, a gap of 1.0 means things are running normally with 0.25
  seconds in between iterations. A gap of 3 indicates that there was
  0.75 seconds between iterations - an an idication that the CPU
  cannot keep up with the work being done in the iterate loop. This
  could, however, be due in part to other processes demanding CPU
  resources.

\item \mvar{PFOOBAR\_ITER\_LEN}: indicates the ratio of (a) observed
  time between the begin and end of the \var{Iterate()} loop, and (b)
  again, the scheduled time between iterations based on the configured
  \psvar{AppTick}. In this case, a value of 1 is an indication that
  the work being done in the iterate loop is dangerously close to
  capacity. 

\end{packed_itemize}
\vspace{0.1in}

\noindent
The \msvar{ITER\_GAP} variable indicates when strain is actually being
exhibited. The \msvar{ITER\_LEN} variable indicates the run-up to
problems.  The \app{uLoadWatch} application monitors the situation by
registering for the \msvar{*\_ITER\_GAP} and \msvar{*\_ITER\_LEN} from
all apps and doing three things:

\begin{packed_enumerate}[0]
\item Threshold detection. A user configured threshold for
  \msvar{ITER\_GAP} values may be set, with a posting to
  \mvar{LOAD\_WARNING} when it is exceeded.

\item An AppCast run warning will also be posted when the user configured
  threshold is exceeded. This will bring a critical load situation to the 
  attention of an operator through the existing AppCast mechanisms.

\item An AppCast report is generated reporting the average and
  maximumn \msvar{ITER\_GAP} and \msvar{ITER\_LEN} noted thus
  far. This can be monitored via one of the AppCast Viewers to monitor
  the load fluctuation.
\end{packed_enumerate}

%====================================================================
\section{Configuration Parameters for uLoadWatch}
%====================================================================
%\index{uLoadWatch!Configuration Parameters}
%\index{Configuration Parameters!uLoadWatch}

The \app{uLoadWatch} application may be configured with a configuration
block within a MOOS mission file, typically with a \var{.moos} file
suffix.  The following parameters are defined for \app{uLoadWatch}.

\vspace{0.15in}
\begin{listing}{Configuration Parameters for uLoadWatch}
\label{lst_phi_params}
\renewcommand{\arraystretch}{1.1}
\begin{longtable}{rp{0.76\textwidth}} 

  \pvar{breach\_trigger}: &amp; The number of times a threshold can be breached 
  before a warning is posted. The default is 1 meaning the first offense is
  forgiven, since it is not uncommon for some apps to have a longer initial 
  gap as they perform one-time start-up work. \\

  \pvar{near\_breach\_thresh}: &amp; In addition to normal threshold
  breaches, the \app{uLoadWatch} app may also monitor for {\em near}
  breaches. A near breach occurs when the normal breach thresh is
  nearly met. For example, if the threshold for a certain app
  is 2.0, this means breach is declared if time between app iterations
  is greater than 2x the normal gap between iterations. If the
  \pvar{near\_breach\_thresh} is set to $0.6$, a near breach is declared
  if the gap reaches $1.6x$ the normal gap. \\

  \pvar{thresh}: &amp; A gap threshold for reporting a
  \mvar{LOAD\_WARNING}. Each threshold line specifies the application
  name and the gap threshold value. \\

\end{longtable}
\end{listing}

%====================================================================
\subsection*{An Example MOOS Configuration Block}

An example MOOS configuration block may be obtained from the command line
with the following:

\begin{consoleverb}
 $ uLoadWatch --example or -e 
\end{consoleverb}

\vspace{0.25in}
\begin{listing}{Example configuration of the uLoadWatch application}
\label{lst_ulw_econfig}
\footnotesize
\begin{verbatim}
  1  =============================================================== 
  2  pHostInfo Example MOOS Configuration                      
  3  ===============================================================
  4                            
  5  ProcessConfig = pHostInfo
  6  {             
  7    AppTick   = 4     
  8    CommsTick = 4    
  9                                                                  
 10    thresh = app=pHelmIvP, gapthresh=1.5
 11    thresh = app=any,      gapthresh=2.0
 12                                                                  
 13    near_breach_thresh = 0.9   // default is off
 14                                                                  
 15    breach_trigger = 5         // default is 1
 16  }                   
\end{verbatim}
\end{listing}
\normalsize
 
%====================================================================
\section{Publications and Subscriptions for uLoadWatch}
%====================================================================
%\index{uLoadWatch!Publications and Subscriptions}
%\index{Publications and Subscriptions!uLoadWatch}
\label{sec_ulw_interface}

The interface for \app{uLoadWatch}, in terms of publications and 
subscriptions, is described below. This same information may also be
obtained from the terminal with:

\begin{consoleverb}
 $ uLoadWatch --interface or -i
\end{consoleverb}

%====================================================================
\subsection{Variables Published by uLoadWatch}
\label{sec_ulw_pubs}

The primary output of \app{uLoadWatch} is the occasional load warning plus
the appcasting report. In Release 20.2, a few additional variables were
added to summarize breaches and near breaches.

\begin{packed_itemize}[2]

\item \mvar{APPCAST}: Contains an appcast report identical to the
  terminal output. Appcasts are posted only after an appcast request
  is received from an appcast viewing utility. 

\item \mvar{LOAD\_WARNING}: A warning indicating an app has been detected
 reporting an gap greater than the configured threshold.

\item \mvar{ULW\_BREACH}: Posted with value \var{true}, if and only when
  a breach has been detected, for any app. 

\item \mvar{ULW\_BREACH\_COUNT}: The total amount of breaches detected,
  for all all apps.
  
\item \mvar{ULW\_BREACH\_LIST}: The list of all apps for which at least
  one breach has been detected.
    
\item \mvar{ULW\_NEAR\_BREACH}: Posted with value \var{true}, if and
  only when a {\em near} breach has been detected, for any app.
  
\item \mvar{ULW\_NEAR\_BREACH\_COUNT}: The total amount of {\em near}
  breaches detected, for all all apps.

\item \mvar{ULW\_NEAR\_BREACH\_LIST}: The list of all apps for which
  at least one {\em near} breach has been detected.
 
\end{packed_itemize}

%====================================================================
\subsection{Variables Subscribed for by uLoadWatch}
\label{sec_ulw_subs}

The \app{uLoadWatch} application subscribes all variables ending with the 
suffix \msvar{\_ITER\_GAP} and the suffix \msvar{\_ITER\_LEN}:

\begin{packed_itemize}[2]
\item \mvar{APPCAST\_REQ}: A request to generate and post a new
  apppcast report, with reporting criteria, and expiration.

\item \mvar{*\_ITER\_GAP}: A report from an application indicating its
  most recent observed gap between iterations.

\item \mvar{*\_ITER\_LEN}: A report from an application indicating its
  most recent observed itertion duration.
\end{packed_itemize}

%====================================================================
\subsection{Command Line Usage of uLoadWatch}
\label{sec_ulw_cmdline}
%\index{Command Line Usage!uLoadWatch}
%\index{uLoadWatch!Command Line Usage}
%\index{Configuration Parameters!uLoadWatch}
%\index{uLoadWatch!Configuration Parameters}

The \app{uLoadWatch} application is typically launched with
\app{pAntler}, along with a group of other shoreside modules. However,
it may be launched separately from the command line. The command line
options may be shown by typing:

\begin{consoleverb}
 $ uLoadWatch --help or -h
\end{consoleverb}

\vspace{0.1in}
\begin{listing}{Command line usage for the uLoadWatch tool}
\label{phi_usage}
\footnotesize
\begin{verbatim}
  1  Usage: uLoadWatch file.moos [OPTIONS]             
  2                                                         
  3  Options:                                               
  4    --alias=&lt;ProcessName&gt;                                
  5        Launch pHostInfo with the given process    
  6        name rather than pHostInfo.                
  7    --example, -e                                        
  8        Display example MOOS configuration block         
  9    --help, -h                                           
 10        Display this help message.                       
 11    --interface, -i                                               
 12        Display MOOS publications and subscriptions.      
 13    --version,-v                                         
 14        Display the release version of pHostInfo.  
 15 
 16  Note: If argv[2] is not of one of the above formats 
 17        this will be interpreted as a run alias. This 
 18        is to support pAntler launching conventions.  
\end{verbatim}
\end{listing}
\normalsize
 
%====================================================================
\section{Usage Scenarios the uLoadWatch Utility}
%====================================================================

A typical usage of the \app{uLoadWatch} utility is shown in Figure
\ref{fig_ulw_topology} below. If the mission is being monitored by a
human (vs. an offline simulation), a load warning will present itself
in any of the appcast viewing tools, e.g., \app{pMarineViewer},
\app{uMACView}, or \app{uMAC}. The \app{pNodeReporter} application
also registers for \mvar{LOAD\_WARNING} mail and will include the load
warning in vehicle node reports. Depending on how \app{pMarineViewer}
is configured, the vehicle label may indicate a load warning if it
occurs.

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.65\textwidth]{figures/ulw_topology.png}
  \caption{{\bf Typical uLoadWatch Topology:} A shoreside or topside
    community is receiving information from several deployed vehicles,
    in the form of node reports, and AppCast reports and warnings. 
    If a load warning is produced, it will (a) show up in any shoreside
    appcast viewer, (b) show up in node reports sent to the shoreside, 
    and (c) show up in local vehicle alog files.}
\label{fig_ulw_topology}
\end{figure}
\vspace{0.1in}

\noindent
When running an offline simulation, the output of \app{uLoadWatch} is
primarily found in the log files. 

%====================================================================
\section{Terminal and AppCast Output}
%====================================================================
\label{sec_ufnb_appcast}

The \app{uLoadWatch} application produces some useful information to
the terminal on every iteration of the application. An example is
shown in Listing \ref{lst_ulw_appcast} below. This application is also
appcast enabled, meaning its reports are published to the MOOSDB and
viewable from any \app{uMAC} application or \app{pMarineViewer}. See
the document for \app{uMac} for more on appcasting and viewing
appcasts.  The counter on the end of line 2 is incremented on each
iteration of \app{uLoadWatch}, and serves a bit as a heartbeat
indicator.  The \var{"0/0"} also on line 2 indicates there are no
configuration or run warnings detected.

\vspace{0.1in}
\begin{listing}{Example terminal or appcast output for \app{uLoadWatch}}
\label{lst_ulw_appcast}
\footnotesize
\begin{verbatim}
  1  ===================================================================
  2  uLoadWatch henry                                           0/0(129)
  3  ===================================================================
  4  Configured Thresholds: 
  5    ANY: 2
  6    PHELMIVP: 1.5
  7  
  8  Application       AvgGap  MaxGap  AvgLen  MaxLen
  9  ----------------  ------  ------  ------  ------
 10  PBASICCONTACTMGR  1.01    1.02    0.00    0.02  
 11  PHELMIVP          1.01    1.04    0.00    0.01  
 12  PHOSTINFO         1.01    1.01    0.00    0.58  
 13  PNODEREPORTER     1.01    1.02    0.00    0.01  
 14  UFLDNODEBROKER    1.01    1.01    0.00    0.00  
 15  ULOADWATCH        1.01    1.01    0.00    0.00  
 16  UPROCESSWATCH     1.00    1.02    0.00    0.00  
 17  USIMMARINE        1.07    1.15    0.02    0.17  
\end{verbatim}
\end{listing}
\normalsize
\vspace{0.1in}

\noindent
The first few lines indicate any user-configured thresholds being
applied to incoming reports. Here a \mvar{LOAD\_WARNING} will be
issued if the \app{pHelmIvP} has an iter gap above 1.5, and if any
application has a gap over 2.0. The remainder of the report (lines
8-17 here) provide a live update of the average and maximum iter gap
and iter length reported for each application reporting.



%====================================================================
\section{How to Modify a MOOS App to Produce Load Information}
%====================================================================
 
All the applications shown in Listing \ref{lst_ulw_appcast} above
produce the \var{*\_ITER\_GAP} and \var{*\_ITER\_LEN} information by
virtue of being an \var{AppCastingMOOSApp}. If you have an application
that does not subclass this superclass, it is still pretty easy to add
this to your class. 

\pskip

To add the \var{ITER\_LEN}, and \var{ITER\_GAP} information, follow
the example of the pseudocode below. The code requires some memory
between iterations of the timestamp of the prior iteration. So the
code requires the additional member variable, added to the
\var{YourApp} class definition, \var{m\_last\_iterate\_time},
initialized to zero in the constructor.

\vspace{0.1in}
\begin{listing}{Adding appcast output for \app{uLoadWatch}}
\label{lst_ulw_appcast_add}
\footnotesize
\begin{verbatim}
  1  YourApp::Iterate() 
  2  {
  3    // Note the time the iterate loop started
  4    double start_time = MOOSTime();
  5 
  6
  7    (The prior guts of your Iterate loop)
  8
  9 
 10    // Note the time the iterate loop ended
 11    double end_time = MOOSTime();
 12 
 13    // Only report iter_gap, iter_len if app frequency is &gt; zero
 14    double app_freq = GetAppFreq();
 15    if(app_freq &gt; 0) {
 16       double interval = 1 / app_freq;
 17       string app_name = MOOSToUpper(GetAppName());89
 18
 19       double iter_len = (MOOSTime() - start_time) / interval;
 20       Notify(app_name + "_ITER_LEN", iter_len);
 21
 22       double iter_gap = (start_time - m_last_iterate_time) / interval;
 23       if(m_last_iterate_time != 0) 
 24         Notify(app_name + "_ITER_GAP", iter_gap);
 25       m_last_iterate_time = start_time;            
 26    }
 27  }
\end{verbatim}
\end{listing}
\normalsize
\vspace{0.1in}

\end{document}

</pre></body></html>