<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{Help Topic: The DULoad Toolbox Rsync Wrappers}
\vspace{0.1in}
\labdate{Spring 2020}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labline
\end{labcover}

\wikidoc{file=Help.DLXRsyncing}
\wikidoc{auth=mikerb@mit.edu}

%====================================================================
\section*{The DULoad Toolbox Rsync Wrappers}
%====================================================================

The DULoad Toolbox is designed to facilitate the use of \cmd{rsync} in
maintaining a remote directory of files. It contains two perl scripts,
one for downloading and one for uploading files, which are essentially
wrappers for \cmd{rsync}. It also utilizes a meta-file on the local
machine to facilitate regular transactions with much less complexity
than the initial transaction. The \cmd{rsync} command is distributed
by default in GNU/Linux and MacOS, and is very powerful, but also
involves a set of lengthy command-line arguments which can be
error-prone. The goals of the DULoad utilities are:

\begin{packed_itemize}
\item Hide the complexities of \cmd{rsync} to make operations easier to
  execute and less error-prone.
\item Implement key metafiles on both the remote and local machines to
  facilitate synchronizing sub-trees.
\item Implement support for masking out "raw" versions of video and
  images.  
\end{packed_itemize}

\pskip

\noindent
The DULoad package is designed for a user who needs access to a
portion of a large data archive, where the whole archive far exceeds
the capacity of the user's local disk storage budget. Typically this
may be an archive with large amounts of video or image files. The user
may want the whole directory structure locally (without files), but
may want the ability to selectively populate portions of the directory
structure with files of current interest. The DULoad package
implements further support for a directory structured with raw and
low-resolution versions of media files. This enables users to download
low-resolution versions as a preview, and only pull-down
higher-resolution files if needed. 

%=====================================================================
\subsection*{Getting the DULoad Package}

The DULoad Toolbox consists of two perl scripts: \cmd{dload.pl} and
\cmd{uload.pl}, and a Bash script \cmd{rrf.sh}. If you are part of the
PavLab group, already with access to the Oceanai server, you can just
check out this tree via SVN:

\vspace{0.15in}
\begin{consoleverb}
 $ svn co svn+ssh://oceanai.mit.edu/svn/repos/project-pavlab
\end{consoleverb}
\vspace{0.15in}

\noindent
The directory \var{project-pavlab/utils/bin} should then be added to
your shell path. 

\noindent
For members outside the lab, the DULoad Toolbox scripts can be download
in a single tar file from here:

\vspace{0.15in}
\begin{consoleverb}
 $ wget http://oceanai.mit.edu/duload/project-duload.tar
\end{consoleverb}
\vspace{0.15in}

\noindent
Once these scripts are added to your shell path, they're ready to go. 

Also see \cite{help_dlx_scheme}.


%=====================================================================
\subsection*{Basic Usage and Working Example}

The duload utilities are designed to be invoked on your local machine,
working with another machine acting as a remote server, e.g., the
\var{oceanai} server used in our lab. It is assumed that you have an
account on the remote server with the ability to \cmd{ssh} onto that
machine (preferably with ssh-keys). We'll use the below directory as
an example. It is a simple event from which two videos were made. The
raw videos are stored, along with a lower-resolution version of each.

\vspace{0.15in}
\begin{consoleverb}
 event_aug1616/
   raw_vids/
     in_water.mov          1.5 GB
     on_shore.mov          1.8 GB
   vids/
     in_water.mp4          181 MB
     on_shore.mp4          472 MB
\end{consoleverb}
\vspace{0.15in}

\noindent
Let's assume it exists on a machine with IP address 18.38.2.158, and on
the local file system as \var{/raiddrive/archives/}. Assuming you have
read access on that machine for this directory, you could copy it onto
your local machine with:

\vspace{0.15in}
\begin{consoleverb}
 $ scp -rp 18.38.2.158:/raiddrive/archives/event_aug1616 .
\end{consoleverb}
\vspace{0.15in}

\noindent
The drawback of \cmd{scp} however is that if this transmission is
interupted before completion, a subsequent \cmd{scp} invocation would
start over again. A better approach with \cmd{rsync} is:

\vspace{0.15in}
\begin{consoleverb}
 $ rsync -aP --partial 18.38.2.158:/raiddrive/archives/event_aug1616 .
\end{consoleverb}
\vspace{0.15in}

\noindent
If the above \cmd{rsync} transmision were interrupted, the subsequent
invocation would only copy files that were not copied on the first
transmission. With the \var{--partial} option, a file interrupted
mid-transmission would resume only to copy the remaining part of the file.


\vspace{0.1in}
\noindent
So \cmd{rsync} is clearly advantageous. The \cmd{dload.pl} utility uses
\cmd{rsync} and implements a few other important features.

%=====================================================================
\subsection*{Basic Usage of the \cmd{dload.pl} utility}

Using the \cmd{dload.pl} utility, an initial synchronization begins with:

\vspace{0.15in}
\begin{consoleverb}
 $ dload.pl --init=18.38.2.158:/raiddrive/archives/event_aug1616
\end{consoleverb}
\vspace{0.15in}

\noindent
This will copy all the remote directories, but not the files:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616 
$ ls
raw_vids/  vids/
\end{consoleverb}
\vspace{0.15in}

\noindent
Once the tree has been initialized locally, we can fill it in later at
our discretion. Even though we have only downloaded the directories,
we can peek at the contents with:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616 
$ dload.pl --all --raw --list-only
drwxrwxr-x        4096 2017/07/10 16:50:32 .
drwxrwxr-x        4096 2017/07/10 16:50:32 raw_vids
-rw-------  1565214170 2017/07/10 16:48:34 raw_vids/in_water.mov
-rw-------  1911673613 2017/07/10 16:49:33 raw_vids/on_shore.mov
drwxrwxr-x        4096 2017/07/10 16:52:53 vids
-rw-r--r--   189222648 2017/07/10 16:51:25 vids/in_water.mp4
-rw-r--r--   494034665 2017/07/10 16:51:15 vids/on_shore.mp4
\end{consoleverb}
\vspace{0.15in}

\noindent
In this example, we may recall that the \var{in\_water} video may
have been interesting. So now that we can see the exact file name, it
can be pulled down for preview with:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616/vids
$ dload.pl in_water.mp4
\end{consoleverb}
\vspace{0.15in}

\noindent
And if it turns out that the high-resolution version is wanted, it can
then be pulled down with:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616/raw_vids
$ dload.pl in_water.mov
\end{consoleverb}
\vspace{0.15in}

\noindent
In practice, it is common to download a directory in its entirety {\em
  except} all the raw files. By default this is the mode \cmd{dload}
operates in.  {\em Raw files are all files who's filename or directory
  name begins with string \var{"raw\_"}, case insensitive}. In our
  example, if the user wanted to download the entire
  \var{event\_aug1616} directory without raw files:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616/
$ dload.pl --all
$ ls *
raw_vids:
vids:
in_water.mp4  on_shore.mp4
\end{consoleverb}
\vspace{0.15in}


%=====================================================================
\subsection*{Using the \var{--dry-run} Command Line Option}

The \cmd{rsync} utility supports a \var{--dry-run} commnand line
switch which shows the user which files {\em would} be transferred if
actually invoked without this option. Returning to our example, in
this case with raw videos not yet downloaded:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616/
$ ls *
raw_vids:                           &lt;-- note raw videos are missing
vids:
in_water.mp4  on_shore.mp4

$ dload.pl --all --raw --dry-run
receiving file list ... 
8 files to consider
raw_vids/                           &lt;-- raw videos *would* be downloaded
raw_vids/in_water.mov
raw_vids/on_shore.mov

sent 98 bytes  received 262 bytes  720.00 bytes/sec
total size is 4175680306  speedup is 11599111.96

$ ls *                  
raw_vids:                           &lt;-- raw videos are still not here
vids:
in_water.mp4  on_shore.mp4
\end{consoleverb}
\vspace{0.15in}

\noindent
The \var{--list-only} switch only lists the directory contents on the
remote machine, and does not show you which remote files are locally
missing, and vice versa. The \var{--dry-run} switch presents you the
{\em differences} between local and remote machines. Using
\var{--dry-run}, or simply \var{-d}, is a good habit to use prior to
most operations, just to confirm that you're actually doing what you
think you're doing.

\vspace{0.1in}

Note: In \cmd{rsync} the \var{--dry-run} option may be abbreviated to \var{-n}.
In the \cmd{dload.pl} and \cmd{uload.pl} utilities, this is abbreviate to \var{-d}.


%=====================================================================
\subsection*{Dowloading Only a Portion of a Directory}

If a portion of a remote archive is all that is needed, this portion
can be just dowloaded directly with the appropriately extended URL. In
our example, if only the \var{vids} are wanted:

\vspace{0.15in}
\begin{consoleverb}
 $ dload.pl --init=18.38.2.158:/raiddrive/archives/event_aug1616/vids 
 $ cd vids
 $ dload.pl --all
\end{consoleverb}
\vspace{0.15in}

\noindent
(In the rare case that you later decide you want a portion of the tree
above the original sub-tree dowloaded, you have three options. (1) The
brute force option is to simply start over by downloading the higher
level tree. This may be fine if the amount of data is relatively small
and bandwidth is high. (2) Download the higher level tree (not yet
with files), and move the previously downloaded files into place. (3)
Create the higher-level parent directories by hand, move the
\var{.rsync\_info} to the root of the tree, and edit this file
accordingly.)

%=====================================================================
\subsection*{Basic Usage of the \cmd{uload.pl} utility}

The \cmd{uload.pl} utility is used for uploading files to tree that has
been previously downloaded via \var{dload.pl}. By example, let's add another
video to our local \var{vids} folder and upload it:


\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616/vids
$ ls
in_water.mp4  on_shore.mp4
$ mv ~/in_air.mp4 .                      &lt;-- move a new video into local folder
$ uload.pl in_air.mp4     
Will handle file: in_air.mp4 
building file list ... 
1 file to consider
in_air.mp4
    15535210 100%    7.00MB/s    0:00:02 (xfer#1, to-check=0/1)

sent 15537234 bytes  received 42 bytes  6214910.40 bytes/sec
total size is 15535210  speedup is 1.00ls
\end{consoleverb}
\vspace{0.15in}

\noindent
The \var{--all} (or \var{-a}) switch can be used to upload everything
in the local directory (including all local directories recursively)
to the remote machine:

\vspace{0.15in}
\begin{consoleverb}
$ uload.pl -a
\end{consoleverb}
\vspace{0.15in}


\noindent
The \var{--user=USER} switch can be used if the local username is different
than the remote username. 

\vspace{0.15in}
\begin{consoleverb}
$ uload.pl --username=jane
\end{consoleverb}
\vspace{0.15in}






%=====================================================================
\subsection*{Local Storage of the \cmd{rsync} Root Location}

A key motivation for implementing the \cmd{dload.pl} and
\cmd{uload.pl} scripts is to ease the user burden on getting complex
command line arguments correct. We really want to be able to confidently
remove any part of the local tree (re-claiming local storage) and get it
back with a trivially simple operation. For example:

\vspace{0.15in}
\begin{consoleverb}
$ cd event_aug1616
$ ls *
  raw_vids:
  in_water.mov  on_shore.mov
  vids:
  in_water.mp4  on_shore.mp4

$ rm -rf raw_vids
$ ls *
  vids:
  in_water.mp4  on_shore.mp4

$ dload.pl -a -r
  raw_vids:
  in_water.mov  on_shore.mov
  vids:
  in_water.mp4  on_shore.mp4
\end{consoleverb}
\vspace{0.15in}

\noindent
Remember the tool is designed for the scenario where the user is working
with a portion of an archive that may be too large to hold in its entirety
on one's local computer. Easily deleting and restoring parts of the tree
is a primary design goal of \cmd{dload.pl}.

\vspace{0.1in}
\noindent
In the above example, after removing the \var{raw\_vids} directory, \cmd{rsync}
could have been used directly to restore the tree with 

\vspace{0.15in}
\begin{consoleverb}
 $ cd event_aug1616
 $ rsync -aP 18.38.2.158:/raiddrive/archives/event_aug1616/raw_vids .
\end{consoleverb}
\vspace{0.15in}

\noindent
Easy, right? You just have to precisely remember the original URL
including the server IP address and the location of the files on the
remote machine. And, if you accidentally put a trailing \var{"/"}, as
in \var{../raw\_vids/}, then you get the contents of \var{raw\_vids} instead of
the directory. The \cmd{dload.pl} utility accomplishes the same with:

\vspace{0.15in}
\begin{consoleverb}
 $ cd event_aug1616
 $ dload.pl -a -r          (-r is short for --raw, -a is short for --all)
\end{consoleverb}
\vspace{0.15in}


\noindent
To accomplish this simplicity, a bit of behind-the-scenes magic needs to
happen. When the (initially empty) tree is first downloaded, a single hidden
file, \var{.rsync\_info}, is created in the root of the tree. This file contains
the "root" of the original tree, URL and all:

\vspace{0.15in}
\begin{consoleverb}
 $ dload.pl --init=18.38.2.158:/raiddrive/archives/event_aug1616
 $ cd event_aug1616
 $ cat .rsync_info
 18.38.2.158:/raiddrive/archives/event_aug1616
\end{consoleverb}
\vspace{0.15in}

\noindent
Whenever \cmd{dload.pl} or \cmd{dload.pl} is invoked, the script initially
looks up the current full path until it finds the \var{.rsync\_info} file.
It uses this information, combined with the full present working directory,
to re-constuct the proper \cmd{rsync} command line argument. Neither
\cmd{dload.pl} or \cmd{uload.pl} will work without it:

\vspace{0.15in}
\begin{consoleverb}
 $ cd event_aug1616
 $ rm -f .rsync_info
 $ dload.pl -a
 The .rsync_info file was not found. Exiting.
\end{consoleverb}
\vspace{0.15in}

\bibliographystyle{plain}
\footnotesize
\bibliography{biblio_help.bib}


\end{document}



</pre></body></html>