<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{uMemWatch: Monitoring App Memory Usage}
\vspace{0.1in}
\labdate{July 2019}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labsrc{project-pavlab/appdocs/app\_umemwatch} 
\labtoc{small}
\end{labcover}

\wikidoc{file=IvPTools.UMemWatch}
\wikidoc{bugz=uMemWatch}
\wikidoc{auth=mikerb@mit.edu}

%=================================================================
\section{Overview}
%=================================================================
\label{sec_umwatch}
\index{uMemWatch}

The \app{uMemWatch} application will monitor the memory used by
selected applications for real time monitoring and post mission
analysis. Typically, without this tool, once a mission has completed,
the memory usage history is lost for all applications. This is
equally true for field robots and simulations. We would like the
ability to launch automated validation simulations that run for
simulated weeks or months, and have confidence that no application
is exhibiting unbounded memory growth, however slow it may be. We
would also like to launch such simulations and be able to detect
and abort the simulations early if memory growth for a particular
app exceeds a pre-defined threshold. In this way we can define a
requirement for an application based on a maximum memory size, for
a mission or class of missions.

%=================================================================
\subsection{An Overview of the Implementation}

Measuring the memory usage of another app, from within
\app{uMemWatch}, is achieved through a system call, with the knowledge
of the system process ID (PID) of the application to be measured.

%-----------------------------------------------------------------
\subsubsection{The System Call for Measuring Memory Usage}

To obtain the memory currently being used by a particular process,
the system command \cmd{ps} is used. If running a simulation where
the PID of \app{pHelmIvP} is \var{37747}, then the following
would reveal the memory usage:

\begin{consoleverb}
$ ps -p 37747
  PID TTY        TIME    CMD
37747 ttys011    0:00.08 pHelmIvP alpha.moos++ pHelmIvP
\end{consoleverb}

\noindent
To get the memory usage, without the header information, the below
switches are used:

\begin{consoleverb}
$ ps -p PID -o rss=
2248
\end{consoleverb}


%-----------------------------------------------------------------
\subsubsection{Obtaining the PID of the Applications to be Measured}
\label{sec_umwatch_getpid}

There are existing command line tools for finding the PID of a process
after it has been launched. For example, the \cmd{pgrep} tool can
be used as follows:

\begin{consoleverb}
$ pgrep pHelmIvP
17223
\end{consoleverb}

\noindent
Armed with this information, we can get the memory usage of
\app{pHelmIvP} as described earlier, using the \cmd{ps} command:

\begin{consoleverb}
$ pgrep pHelmIvP
17223
$ ps -p 17223 -o rss=
2248  
\end{consoleverb}

\noindent
However, in multi-vehicle missions on a single computer, there are
likely multiple versions of the same app running, one per simulated
vehicle. A query for the PID of an app may result in something like:

\begin{consoleverb}
$ pgrep pHelmIvP
17811
17817
17824
17829
\end{consoleverb}

\noindent
The intended use of \app{uMemWatch} is to have one instance of
\app{uMemWatch} running per MOOS community and monitor the apps in
that community only. To accomplish this, if there are multiple
versions of an app running in a simulation, \app{uMemWatch} needs to
know which app is associated with its own community. This is
accomplished by having each app publish its PID in a dedicated MOOS
variable. For example, the PID of \app{pMyApp} is published in the
variable \mvar{PMYAPP\_PID}. If \app{pMyApp} is being monitored
by \app{uMemWatch}, then \app{uMemWatch} will subscribe for this
variable and to learn the PID of \app{pMyApp} and use the PID
in subsequent system calls to measure the memory.







%=================================================================
\subsection{Ensuring Reasonable Monitoring}

In the spirit of Heisenberg, we recognize that the process of
monitoring system resource consumption is itself a consumption of
system resources. Each measurement system call is an invocation of the
\cmd{ps} command, which in itself results in a new PID for each call.
Our goal is to ensure a pace of measurement that provides us with the
information we need, but using the smallest footprint required.  To
some extent we let the user decide some of these choices through
configuration parameters, using reasonable default values.


%-----------------------------------------------------------------
\subsubsection{Controlling the Application Watch List}
\label{sec_umwatch_list}

By default, the \app{uMemWatch} {\em watch list} will include any
other application connected to the MOOSDB, as known through the
\mvar{DB\_CLIENTS} list. This doesn't necessarily mean these apps will
be measured, since the apps themselves must be enabled to detect their
own PID and post it to the MOOSDB. For such apps, a convention is used
to post the PID information upon app startup, based on the name. For
example the app \app{pHelmIvP} will post its PID as the contents of
the MOOS variable \mvar{PHELMIVP\_PID}. And the app
\app{pContactMgrV20} will post its PID as the contents of the MOOS
variable \mvar{PCONTACTMGRV20\_PID}, and so on. See Section
\ref{sec_umwatch_get_pid}.

\vspace{0.1in}
\noindent
Again, this is only true for MOOS apps implemented to follow this
convention. As of this writing, this convention is supported by the
following apps: \app{pHelmIvP}, \app{uTimerScript},
\app{pNodeReporter}, \app{pBasicContactMgr}, \app{pContactMgrV20},
\app{uMonitorCPU}, \app{uFldNodeBroker}, \app{pMarineViewer}.

\vspace{0.1in}
\noindent
The \app{uMemWatch} app may be configured to ignore certain apps it
does not need to measure, by using the configuration variable:

\prevblue
\begin{verbatim}   
  ignore = pNodeReporter, uFldNodeBroker
\end{verbatim}
\postverb

\noindent
Alternatively, the app may be configured to only watch certain apps it
needs to measure, by using the configuration variable:

\prevblue
\begin{verbatim}   
  watch_only = pHelmIvP, pBasicContactMgr
  watch_only = pNodeReporter
\end{verbatim}
\postverb

\noindent
A configuration warning will be generated if a watch item is given
that was previously specified above as an ignore item, and the watch
item configuration will be ignored. The same is true in the opposite
direction.

%-----------------------------------------------------------------
\subsubsection{Controlling the Pace Irrespective of Time Warp}
\label{sec_umwatch_pace}

As described above, \app{uMemWatch} will run with a {\em watch list}
of applications. On any given iteration of the \app{uMemWatch} Iterate
loop, at most only one application will be have its current memory
usage measured. Successive measurements will cycle through the
applications on the watch list. This ensures there are never multiple
simultaneous invocations of a system command for querying memory
usage, and that memory queries are spread equally across all
applications on the watch list.
 
\vspace{0.1in}
\noindent
Timing each system call measurement to the \app{uMemWatch} Iterate
loop is a bit perilous if the user decides to use a high AppTick
or if \app{uMemWatch} is used in a simulation with a high time warp.
For this reason, the frequency of measurements is limited to
at most once per second, always measured in real time, ignoring
time warp. This frequency is set with the following configuration
parameter:

\prevblue
\begin{verbatim}   
  absolute_time_gap = 5
\end{verbatim}
\postverb

\noindent
Configuration of the above parameter to a value below 1 will result in
a configuration warning and the resulting setting will be set to 1. To
recap by example, using the above \pvar{absolute\_time\_gap=5}, and if
\app{uMemWatch} were watching three applications in simulation with
time warp of 40, each application would be measured for memory usage,
once every 15 seconds, or once every 600 iterations of \app{uMemWatch}.

\noindent
Since the intended use of \app{uMemWatch} is to monitor longer term
trends, and detect unbounded growth, measuring frequently is probably
not needed. Furthermore, since a system call is used for this
measurement, we are extra conservative about not doing this more
frequently than intended. The default value for
\pvar{absolute\_time\_gap} is 5 seconds.



%===========================================================
\section{Configuration Parameters for uMemWatch}
%===========================================================
\index{uMemWatch!Configuration Parameters}
\index{Configuration Parameters!uMemWatch}

The following parameters are defined for \app{uMemWatch}. A more
detailed description is provided in other parts of this
section. Parameters having default values are indicated.

\vspace{0.15in}
\begin{listing}{Configuration Parameters for \app{uMemWatch}}
\label{lst_params_umwatch}
\renewcommand{\arraystretch}{1.1}
\begin{longtable}{rp{0.76\textwidth}} 

  \pvar{absolute\_time\_gap}: &amp; The time, in seconds, between system
  calls for measuring memory. This is unaffected by time warp. The
  default is 5 seconds. Section \ref{sec_umwatch_pace}.\\

  \pvar{ignore}: &amp; Comma separated list of apps to not watch. Multiple
  lines ok. Section \ref{sec_umwatch_list}. \\

  \pvar{watch\_only}: &amp; Comma separated list of apps to 
  watch. Multiple lines ok. Section \ref{sec_umwatch_list}. \\

\end{longtable}
\end{listing}
  

%=================================================================
\section{Publications and Subscriptions of uMemWatch}
%=================================================================
\index{uMemWatch!Publications and Subscriptions}
\index{Publications and Subscriptions!uMemWatch}
\label{sec_umwatch_interface}

The interface for \app{uMemWatch}, in terms of publications and
subscriptions, is described below. This same information may also be
obtained from the terminal with:

\vspace{0.1in}
\begin{consoleverb}
 $ uMemWatch --interface or -i
\end{consoleverb}
\vspace{0.05in}

%=================================================================
\subsection{Variables Published by uMemWatch}
\label{sec_umwatch_pubs}

The only output of \app{uMemWatch} is: 

\begin{packed_itemize}[2]
\item \mvar{APPCAST}: Contains an appcast report identical to the
  terminal output. Appcasts are posted only after an appcast request
  is received from an appcast viewing utility. Section
  \ref{sec_umwatch_appcast}.

\item \mvar{APPNAME\_MEM}: The amount of memory currently measured
  for the given applicion. This is a numerical value in Megabytes.
\end{packed_itemize}

%=================================================================
\subsection{Variables Subscribed for by uMemWatch}
\label{sec_umwatch_subs}

The \app{uMemWatch} application will subscribe for the following
four MOOS variables:

\begin{packed_itemize}[2]
\item \mvar{APPCAST\_REQ}: A request to generate and post a new
  apppcast report, with reporting criteria, and expiration.

\item \mvar{DB\_CLIENTS}: A list of clients (apps) known the
  local MOOSDB community. 

\item \mvar{APPNAME\_PID}: 
  \var{"ignore"} (\var{"none"}). Section \ref{sec_umwatch_getpid}.

\end{packed_itemize}


%=================================================================
\section{Terminal and AppCast Output}
%=================================================================
\label{sec_umwatch_appcast}

The \app{uMemWatch} application produces some useful information to
the terminal on every iteration of the application. An example is
shown in Listing \ref{lst_umwatch_appcast} below. This application is
also appcast enabled, meaning its reports are published to the MOOSDB
and viewable from any \app{uMAC} application or
\app{pMarineViewer}. See Section \ref{sec_umac_utils} for more on
appcasting and viewing appcasts.  The counter on the end of line 2 is
incremented on each iteration of \app{uMemWatch}, and serves a bit as
a heartbeat indicator.  The \var{"0/0"} also on line 2 indicates there
are no configuration or run warnings detected.

\pskip

\noindent
The output in the below example comes from the example described in 
Section \ref{sec_umwatch_example}. 

\vspace{0.1in}
\begin{listing}{Example terminal or appcast output for \app{uMemWatch}}
\label{lst_umwatch_appcast}
\footnotesize
\begin{verbatim}
 1  ===================================================================
 2  uMemWatch alpha                                            0/0(394)
 3  ===================================================================
 4  Configuration:     
 5    Watch Only Apps: pHelmIvP,pMarineViewer
 6        Ignore Apps: None
 7  Absolute Time Gap: 1
 8
 9 
10  App            PID    Memory  Delta 
11  -------------  -----  ------  ------
12  pHelmIvP       35346  1.988   0.008 
13  pMarineViewer  35347  95.492  -0.016
14  
15   Last ix: 0
16  Last app: pHelmIvP
17  Last mem: 1.988
18  Last raw: 1988
19  Warp: 10
\end{verbatim}
\end{listing}
\normalsize
\vspace{0.1in}

\noindent
The first few lines (4-7) show the configuration settings for
\app{uMemWatch}.  The status of \app{uMemWatch} is shown in Lines
10-19. The watch list is comprised of \app{pHelmIvP} and
\app{pMarineViewer}, and their status is shown on lines 12 and 13. The
system process ID (PID) is shown in the second column.  The latest
measure of memory usage is shown in the third column, and the change
in memory usage since the previous measurement is shown in the last
column. Line 15 shows the index of the most recently measured
application, and line 16 shows the name of the app at this index. As
the mission unfolds one can see this index cycling from 0 to N-1 where
N is the number of processes being monitorer. This cycling through
processes was discussed in Section \ref{umwatch_pace} as way to ensure
infrequent system calls. The latest memory measurement is shown
in lines 17-18. Line 17 shows the memory usage in Megabytes, and
line 18 shows the raw output from the system command. Line 19
simply confirms that time warp detected by \app{uMemWatch}.

%=================================================================
\section{A Simple Example}
%=================================================================
\label{sec_umwatch_example}

The below is a simple example configuration of \app{uMemWatch}.
In this case only two processes are watched, \app{pHelmIvP} and
\app{pMarineViewer}. This example configuration may also be
obtained by typing on the command line: \var{uMemWatch -e}.

\vspace{0.1in}
\begin{listing}{A Simple \app{uMemWatch} Example}
\label{lst_umwatch_example}
\footnotesize
\begin{verbatim}
//------------------------------------------
// uMemWatch config block

ProcessConfig = uMemWatch
{
  AppTick   = 4
  CommsTick = 4

  absolute_time_gap = 1   // In Seconds, Default is 4
                                                     		
  watch_only = pHelmIvP,pMarineViewer
}

\end{verbatim}
\end{listing}
\normalsize

\end{document}

</pre></body></html>