\subsubsection{Problem}
We wish to send periodic updates of CTD data from an AUV using a WHOI Micro-Modem. The Micro-Modem
currently supports three packet sizes (32 byte FSK packet and two higher rate PSK packets (64 byte and 256 byte)).
the goal is to package this CTD data as compactly as possible in a manner that allows for as many
CTD samples to be sent in a packet as possible.

\subsubsection{Solution}
pCTDCodec\footnote{codec = \textbf{co}mpression / \textbf{dec}ompression. While the term is often used for compression/decompression of stream data (think video/audio) we will coopt is here in a more general sense.} performs the following tasks:
\begin{itemize}
\item encodes CTD (specifically: salinity, temperature, depth, time,
lat, lon) information into hexadecimal sentences for sending through any rate limited channel (primarily the WHOI acoustic micromodem).
\item decodes the incoming hex into human readable messages.
\item computes the sound speed using the algorithm given in
(K.V. Mackenzie, \textit{Nine-term equation for the sound speed in the oceans} (1981) J. Acoust. Soc. Am. 70(3), pp 807-812).
\end{itemize}

\subsubsection{Background}

pCTDCodec generates a hexadecimal message that gets ``encoded'' (really just wrapped) by pAcommsHandler (using DCCL). Since pAcommsHandler (DCCL) adds two bytes to the message (one byte for CCL and the other byte for pAcommsHandler "MOOS routing") we have 30, 62, or 254 bytes to use for a given FSK, rate 2 PSK, or rate 3/5 PSK packet, respectively.

\subsubsection{Usage}
\paragraph{Example moos file}
pCTDCodec must be run on all sending and receiving nodes. Here's an example .moos file (used by BF21 AUV 'Unicorn' at the GLINT08 sea trials):

\begin{small}\begin{verbatim}
ProcessConfig = pCTDCodec
{
  // available to all moos processes.
  AppTick    = 4
  CommsTick  = 4

  // available to all tes moos processes

  //verbose, terse, quiet
  verbosity = verbose

  // all case insensitive

  publish_CTD = true

  ////////////////////////////////
  //         OUTGOING           //
  ////////////////////////////////

  // omission of any will cause no writes for that size
  outgoing_254_byte_var = OUT_CTD_HEX_254B
  outgoing_62_byte_var = OUT_CTD_HEX_62B
  outgoing_30_byte_var = OUT_CTD_HEX_30B
  // publish this for error checking and for wifi sending, if desired
  outgoing_human_readable_var_254 = OUT_CTD_254B
  outgoing_human_readable_var_62 = OUT_CTD_62B
  outgoing_human_readable_var_30 = OUT_CTD_30B

  // time between publishes in seconds.
  // samples taken evenly for this time frame.
  // for example, if publish_delta is 120 and we can 
  // fit 3 messages in a frame, one will be taken from 40 sec,
  // another from 80 sec and a third from 120 sec
  // since the last publish
  publish_delta = 30

  ////////////////////////////////
  //         INCOMING           //
  ////////////////////////////////

  incoming_254_byte_var = IN_CTD_HEX_254B
  incoming_62_byte_var = IN_CTD_HEX_62B
  incoming_30_byte_var = IN_CTD_HEX_30B

  incoming_human_readable_var_254 = IN_CTD_254B_DEC
  incoming_human_readable_var_62 = IN_CTD_62B_DEC
  incoming_human_readable_var_30 = IN_CTD_30B_DEC

  ////////////////////////////////
  //    FRAMING PARAMETERS      //
  ////////////////////////////////

  time_precision = 0

  // where we grab depth from
  depth_var = NAV_DEPTH
  // range of depth values we expect
  // (greater or less will be set to the respective limit)
  // keeping this range tight allows better data throughput
  depth_range = 0, 100
  // change in depth required to produce new keyframe - 
  // too tight and you will drop frames, too loose and you waste data
  depth_delta = 10
  // number of decimal places of precision required
  depth_precision = 3

  temp_var = CTD_TEMPERATURE
  temp_range = 10, 30
  temp_delta = 3.0
  temp_precision = 3

  sal_var = CTD_SALINITY
  sal_range = 30, 40
  sal_delta = 0.5
  sal_precision = 3

  lat_var = NAV_LAT
  lat_range = 42, 43
  lat_delta = 0.001
  lat_precision = 5

  lon_var = NAV_LONG
  lon_range = 9, 11
  lon_delta = 0.02
  lon_precision = 5
}

\end{verbatim}\end{small}

\paragraph{Filling out the .moos file: picking ranges, precisions and deltas}

Here we will go through all the parameters allowed for the .moos file. The concepts of "range", "delta", and "precision" will need the most explanation.

\paragraph{General parameters}
\begin{itemize}
\item \verb|verbosity|: choose verbose for full text terminal output, terse for symbolic heartbeat output, and quiet for no terminal output.
\item \verb|publish_CTD|: boolean (true/false) that specifies if pCTDCodec should publish ANY outgoing CTD information. should be set true for any vehicle with a CTD and false for a receiving only node (such as a topside computer). defaults to true if omitted.
\end{itemize}

\paragraph{Outgoing parameters}
These values govern the encoding behavior of the codec:
\begin{itemize}
\item \verb|outgoing_254_byte_var|: name of the MOOS variable to publish outgoing (compressed) the 254 byte hexadecimal data. if omitted, the 254 byte message will not be published. 
\item \verb|outgoing_62_byte_var|: similar, but for the 62 byte message.
\item \verb|outgoing_30_byte_var|: similar, but for the 30 byte message.
\item \verb|outgoing_human_readable_var_254|: name of the MOOS variable to publish a copy of the outgoing human readable samples. these messages will match the decoded messages placed in \verb|incoming_human_readable_var_254| by the receiving vehicle. If omitted, these message will not be published.
\item \verb|outgoing_human_readable_var_62|: similar, but for the 62 byte message.
\item \verb|outgoing_human_readable_var_30|: similar, but for the 30 byte message.
\item \verb|publish_delta|: how often (in seconds) to produce a message of each type. It is advisable to set this value larger than the time between sending messages. For example, in a MAC scheme allowing for sending from this vehicle every minute at FSK rate 1 (a single 32 byte message), you should set \verb|publish_delta| to at least 60, perhaps several times higher depending on what other messages are in the queue (status, etc.). Samples are taken at even time intervals within the \verb|publish_delta| interval. For example, if the message can hold 3 samples and \verb|publish_delta| is 30 (seconds), then a sample for the message is taken every 10 seconds. Clearly, a larger message (e.g. 254 byte) will sample at a higher rate than a smaller message (e.g. 30 byte) in order to fill the message.
\end{itemize}

\paragraph{Incoming parameters}
These values govern the decoding behavior of the codec:
\begin{itemize}
\item \verb|incoming_254_byte_var|: name of the MOOS variable to subscribe for to look for incoming 254 byte hexadecimal messages. if you are using pAcommsHandler, this should match the name in the \verb|receive=| field. if omitted, the pCTDCodec will not look for an incoming 254 byte message.
\item \verb|incoming_62_byte_var|: similar, but for the 62 byte message.
\item \verb|incoming_30_byte_var|: similar, but for the 30 byte message.
\item \verb|incoming_human_readable_var_254|: name of the MOOS variable to publish the decoded samples. Each sample will be published as a separate string, so for a large message, this variable may get a large number of rapid publishes in succession.
\item \verb|incoming_human_readable_var_62|: similar, but for the 62 byte message.
\item \verb|incoming_human_readable_var_30|: similar, but for the 30 byte message.
\end{itemize}

\paragraph{Framing parameters} these values govern the way the message is designed. Thus, they determine how much space each data value takes in the message.
\begin{itemize}
\item \verb|time_precision|: number of decimal places required for the timestamp sent with the message (which is UTC seconds since 1/1/70). e.g. \verb|time_precision=1| means to round the time to the nearest tenth second. the smaller this value, the less space storing the time takes in the message. so balance quantity of messages with your tolerance for precision. time is pulled from the vehicle clock using the MOOS function call \verb|MOOSTime()|.
\item \verb|depth_var|: the MOOS variable (double type) that contains the depth of the CTD instrument in meters.
\item \verb|depth_range|: a comma separated field \verb|depth_range=(double)min,(double)max| specifying the absolute minimum and maximum values for depth the vehicle is expected to experience. if the value of \verb|depth_range| is ever smaller or larger than the min or max specified here, the depth will be fixed to the respective limit. for example, a vehicle operating in shallow water (max depth 180 meters) could have \verb|depth_range=0,180|. it is the responsible of the user to detect saturation at either limit. thus, it is prudent perhaps to use limits that safely exceed the operational parameters (maybe \verb|depth_range=-10,200| for the above example). obviously, the smaller the range, the fewer bits required to encode the depth in the message. 
\item \verb|depth_delta|: this is the trickiest parameter to understand. bear with me, and perhaps read the encoding details that follow in the document. every hex message contains several CTD samples (a single look at T,S,D,lat,lon,time). the first sample (key frame) is sent "in full" (though the ranges are taken into account). all remaining samples are sent as offsets ("deltas") to this first sample (key frame). this is done to maximize the number of samples in the message. however, the size (in bits) of each field in the message must be calculable by the receiver (or we waste space specifying the layout IN each message). thus, in order to know the size of the delta frames, we must specify maximum deviations from the key frame we expect to experience in the time between messages. that is, the maximum depth change we expect to experience in \verb|publish_delta| seconds should be given here. if the delta is exceeded, the message is discarded by the decoding unit of the receiving pCTDCodec. thus, bad data is not a concern here, but waste is. an example: our vehicle can dive at 0.5 m/s. our \verb|publish_delta=30| (a message is generated every 30 seconds). thus, we will not experience more than $\pm$ 15 meters change within the span of a message. thus, we should set \verb|depth_delta=15|.
\item \verb|depth_precision|: how many decimal places to send with the depth message. e.g., \verb|depth_precision=2| means round to the nearest centimeter (10.2563 meters is sent as 10.26).
\end{itemize}

The remaining parameters are given in the above example .moos file and follow the exact same idea as those given for depth. All units are assumed SI (or no units in the case of salinity). \textbf{Make sure you're running the same moos configuration for all the FRAMING PARAMETERS on both the vehicle and topside. If not, you will incorrectly decode messages!}

\paragraph{Input formats}

The values in the moos variables given by \verb|depth_var|,  \verb|sal_var|, \verb|lat_var|, and \verb|lon_var| should be double precision numbers. e.g. if \verb|depth_var=NAV_DEPTH|, the moos variable should look like \verb|NAV_DEPTH=20.35962|

\paragraph{Output formats}
This process produces two types of strings:
\begin{enumerate}
\item a hexadecimal string suitable for sending through the acoustic modem (ASCII characters [a-e$|$0-9]). for example, if \verb|outgoing_254_byte_var=OUT_CTD_HEX_254B|, the string produced will look a bit like:
\begin{verbatim}
OUT_CTD_HEX_254B: 6850a232ccd2be62e1f36e6b02fff...
\end{verbatim}
\item a human readable string suitable for easy processing by others. a string is produced for each sample. for example, if \verb|incoming_human_readable_var_254=IN_CTD_254B_DEC|, the string produced will look a bit like:
\begin{verbatim}
IN_CTD_254B_DEC: originator=29,time=1217485957,depth=0.200,temp=26.122,
sal=39.328,lat=42.58190,lon=10.10876,mackenzie_soundspeed=1541.56930
\end{verbatim}
where \verb|originator| is the sending vehicle's modem id and \verb|time| is given in seconds since 1/1/1970 (unix epoch time). the rest of the parameters are the in the standard SI units (or unitless for salinity).

\end{enumerate} 

\paragraph{Further usage notes}
It is not important to understand the codec scheme to properly use this process. You will
want to run pCTDCodec on each vehicle with a CTD and also on topside to decode messages.
Simply fill out the sample .moos block with the correct parameters (use precisions from the CTD
instrument / GPS, most likely). The tighter you can make the ranges and the deltas, the
better (in terms of messages / packet). If you want to see how many frames you'll get per packet
just run pCTDCodec and watch the terminal output (in verbose mode) at startup.

It's worth noting this again here: \textbf{please make sure you have the same framing parameters on both the vehicle and topside.
if not, messages will be wrongly decoded without warning.}


\subsubsection{Configuring pAcommsHandler for pCTDCodec}

Add the following lines to the pAcommsHandler .moos file:
\begin{small}\begin{verbatim}
message_file = /path/to/pCTDCodec_32B.xml
message_file = /path/to/pCTDCodec_64B.xml
message_file = /path/to/pCTDCodec_256B.xml
\end{verbatim}\end{small}

pCTDCodec\_32B.xml:
\begin{small}\begin{verbatim}
<?xml version="1.0" encoding="UTF-8"?>
<message_set>
  <message>
    <name>pCTDCodec_32B</name>
    <id>1</id>
    <trigger>publish</trigger>
    <trigger_moos_var>OUT_CTD_HEX_30B</trigger_moos_var>
    <size>32</size>
    <layout>
      <hex>
        <name>data</name>
        <num_bytes>30</num_bytes>
      </hex>
    </layout>
    <on_receipt>
      <publish>
        <moos_var>IN_CTD_HEX_30B</moos_var>
        <all />
      </publish>
    </on_receipt>
    <queuing>
      <ack>false</ack>
      <blackout_time>0</blackout_time>
      <max_queue>100</max_queue>
      <newest_first>true</newest_first>
      <priority_base>0.28</priority_base>
      <priority_time_const>120</priority_time_const>
    </queuing>
  </message>
</message_set>
\end{verbatim}\end{small}

pCTDCodec\_64B.xml:
\begin{small}\begin{verbatim}
<?xml version="1.0" encoding="UTF-8"?>
<message_set>
  <message>
    <name>pCTDCodec_64B</name>
    <id>2</id>
    <trigger>publish</trigger>
    <trigger_moos_var>OUT_CTD_HEX_62B</trigger_moos_var>
    <size>64</size>
    <layout>
      <hex>
        <name>data</name>
        <num_bytes>62</num_bytes>
      </hex>
    </layout>
    <on_receipt>
      <publish>
        <moos_var>IN_CTD_HEX_62B</moos_var>
        <all />
      </publish>
    </on_receipt>
    <queuing>
      <ack>false</ack>
      <blackout_time>0</blackout_time>
      <max_queue>100</max_queue>
      <newest_first>true</newest_first>
      <priority_base>0.29</priority_base>
      <priority_time_const>120</priority_time_const>
    </queuing>
  </message>
</message_set>
\end{verbatim}\end{small}

pCTDCodec\_256B.xml:
\begin{small}\begin{verbatim}
<?xml version="1.0" encoding="UTF-8"?>
<message_set>
  <message>
    <name>pCTDCodec_256B</name>
    <id>3</id>
    <trigger>publish</trigger>
    <trigger_moos_var>OUT_CTD_HEX_254B</trigger_moos_var>
    <size>256</size>
    <layout>
      <hex>
        <name>data</name>
        <num_bytes>254</num_bytes>
      </hex>
    </layout>
    <on_receipt>
      <publish>
        <moos_var>IN_CTD_HEX_254B</moos_var>
        <all />
      </publish>
    </on_receipt>
    <queuing>
      <ack>false</ack>
      <blackout_time>0</blackout_time>
      <max_queue>100</max_queue>
      <newest_first>true</newest_first>
      <priority_base>0.30</priority_base>
      <priority_time_const>120</priority_time_const>
    </queuing>
  </message>
</message_set>
\end{verbatim}\end{small}

\subsubsection{Details}
Here are details that are probably not crucial to the end user of this codec, but are of interest to the advanced
user or curious traveler.
\paragraph{Sample rates, packing, etc.}
On startup, pCTDCodec uses the parameters given to determine how many CTD message
snapshots (from now on, ``frames'') we can fit into a given 30 byte packet (call it $f_{30}$) and 62 or 254 byte packet (call them $f_{62}$ and $f_{254}$).
This value is based on the ranges and precisions given in the .moos file (more on this later).
Furthermore, a value is given (\verb|publish_delta| in the .moos file) for how often (call it $N$) pCTDCodec should publish a packet.

Thus, to publish $f_{30}$ frames every $N$ seconds we must take a snapshot every $N/f_{30}$ seconds.

pCTDCodec subscribes constantly for double MOOS variables containing latitude, longitude,
temperature, salinity, and depth and stores the newest of all these values. When it is time to take snapshot
(every $N/f_{30}$ seconds for the 30 byte packet (and similarly for the two larger packets), the most current
values for all these parameters are pushed to a stack.

Every $N$ seconds, the stacks are flushed, messages are assembled, encoded, and published to the appropriate variables
(set in .moos parameters: outgoing\_30\_byte\_var, outgoing\_62\_byte\_var, outgoing\_254\_byte\_var).

Any guarantee of receipt is handled by pAcommsHandler.
	
\paragraph{Message structure}
\begin{figure}
\centering
\includegraphics[scale=.5]{ctd_packet_diagram.eps}
\caption{Schematic of pCTDCodec encoding/decoding}
\label{packet_diagram}
\end{figure}


Here's where the fun is. Read on if you want to know the details of how this process codes and decodes messages.
Let us examine the 30 byte message, as the 62 byte and 254 byte messages are the same, 
just longer. The meta structure of the message is:

(left to right as you read the hexadecimal message) \\

[key frame][delta frame][delta frame][delta frame] ... 

The key frame can be thought of as a ``complete'' CTD message which, when paired with the information in the 
configuration file ("global key"), is sufficient to give all the data. Each delta frame is stored as a difference
from the key frame. Reference figure \ref{packet_diagram} throughout this discussion for a pictorial representation.

For example, say i had the following three messages to send (ignoring lat, lon for now):

time=10,temp=15,sal=25,depth=100\\
time=11,temp=14,sal=23,depth=103\\
time=12,temp=16,sal=25,depth=102\\

The key frame would be \\
time=10,temp=15,sal=25,depth=100

and the delta frames would be \\
time=1,temp=-1,sal=-2,depth=3 \\
time=2,temp=1,sal=0,depth=2

By sending deltas we save space, especially for parameters that tend to stay relatively constant within
the packet interval $N$ (lat / lon) or have a predictable delta (time, where the delta will never
be more than $N$ and never negative).

To further confuse matters, but make the messages small, is that each key frame is itself
keyed to another source, the configuration file (or .moos file, which can be thought of as a "global key"). For every parameter (temp, sal, depth, lat, lon) the user can specify a range (minimum value, maximum value) and a delta. The min and max values
should be set to the extremes expected to be seen for those values in the given experiment at hand.
(For example, if operating in a small part of italy, you can set pretty narrow bounds on lat, lon compared to
 [-90, 90], [-180, 180]). The key frame is encoded and decoded based on these values. If the values fall outside the range
for some reason (don't let this happen by careful choice of parameters) pCTDCodec fixes it to the respective limit (no overflows are allowed).

Now, the deltas set in the configuration file should be how much you expect the given parameter
to vary in the publish\_delta interval. if you set publish\_delta to 120 seconds, set the depth\_delta
to whatever you expect your vehicle (in the extreme) to be able to go up or down in 120 seconds. If deltas
are exceeded, the messages are dropped, and you will see a warning on your topside version of pCTDCodec.

Finally, for each variable you set a precision you need for the value. This precision is decimal places, so
temp\_precision=3 means preserve three decimal places for temperature (10.2346245... will be sent as 10.235). Set precisions
as low as you can to pack more into a message. rounding is done as follows:

(for precision=0) \\
1.5 goes to 2 \\
2.5 goes to 2 (note this!)\\ 
1.567 goes to 2 \\
1.4642 goes to 1 \\
2.4 goes to 2 \\
2.5632 goes to 3 

This removes the ``upward'' bias caused if you were to always round 5 up. Instead,
we round 5 to even.

Back to the message structure: [key frame][delta frame][delta frame][delta frame] ... 

Within each frame we have pieces so:

[key frame] is broken into: 

[(key\_time)(key\_depth)(key\_temp)(key\_sal)(key\_lat)(key\_lon)]

and similarly for delta frames.

Key frames: the size (in bits) of each piece is set at run time by the configuration parameters. To understand
how big each piece must be, we need to understand the encoding scheme used. Essentially, each
value is turned into an unsigned int by multiplying by $10^{precision}$, rounding (by the method above)
to the nearest int and subtracting the minimum of the range. we will use $val$ to represent
any of the variables (time, depth, temperature, salinity, latitude, longitude).

Specifically:
\begin{equation}
 val_{key,encoded} = \hbox{round}((val_{key} - val_{min})*10^{precision})
\end{equation}

Decoding reverses this:
\begin{equation}
 val_{key,decoded} = val_{key,encoded}/10^{precision} + val_{min}
\end{equation}

As you no doubt have realized, the configuration blocks for the encoding process must
match those for the decoding process. There is no reason, however, multiple copies of
pCTDCodec couldn't be run topside with different configurations to decode messages
from different vehicles.

Delta frames encoding:
\begin{equation}
 val_{delta,encoded} = \hbox{round}((val_{delta}-val_{key} + delta)*10^{precision})
\end{equation}

Decoding:
\begin{equation}
 val_{delta,decoded} = val_{delta,encoded}/10^{precision} + val_{key} - delta
\end{equation}
where $delta$ is the (almost) the value specified by $val$\_delta in the configuration block.

Time is the only parameter not given ranges for in the .moos block. This is because time is forced
to be unix epoch and is given time\_range = start\_of\_day, start\_of\_day + 86400, and time\_delta = publish\_delta

The size (in bits) of each piece is determined by
\begin{equation}
 size_{key\_piece} = \hbox{ceil}(\log_2((val_{max} - val_{min})*10^{precision} + 1))
\end{equation}
and 
\begin{equation}
 size_{delta\_piece} = \hbox{ceil}(\log_2((2*delta)*10^{precision}) + 1)
\end{equation}

Twice $delta$ is used as the delta can be positive or negative (time\_delta = 0.5 means time can be $\pm 0.5$). The additional ``+1'' compensates for the fact you have to store "0". For example, 4 bits can store 16 values, but only a maximum value of 15.
The size of any given piece cannot exceed the length of an unsigned long on your system (typically 32 bits, 4 bytes). Be careful of this.

Finally, putting it all together, our binary message is (lsb = least significant bit, msb = most significant bit):

[(msb of key\_time \ldots lsb of key\_time)(msb of key\_depth \ldots lsb of key\_depth) \ldots ][(msb of delta1\_time \ldots lsb of delta1\_time)(msb of delta1\_depth \ldots lsb of delta1\_depth) \ldots ] \ldots [\ldots (msb of delta(N-1)\_lon \ldots lsb of delta(N-1)\_lon][(zero padding to fill message)]

Then, the binary message is split into nibbles (4 bits) and each is written as a hex character to form the final sent message.
