<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{Help Topic: The rrf.sh Utility}
\vspace{0.1in}
\labdate{Spring 2020}
\vspace{0.1in}
\labauth{Michael Benjamin, mikerb@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labline
\end{labcover}

\wikidoc{file=Help.DLXRemoving}
\wikidoc{auth=mikerb@mit.edu}

%====================================================================
\section*{The rrf.sh Utility Overview}
%====================================================================

The \cmd{rrf.sh} (remove raw files) utility is designed to easily
remove a set of {\em raw} files on a local file system, presumably as
the user is bumping up against the limits of local drive capacity. The
\cmd{rrf.sh} utility is part of the Duload Toolbox which also includes
the \cmd{dload.pl} and \cmd{uload.pl} utilities.

\vspace{0.15in}
\noindent
The \cmd{rrf.sh} utility is a Bash script, thus the \var{.sh} suffix.
This utility should be in your own shell path. You don't need to be
running the Bash shell as your terminal shell. The first line of the
script is:

\vspace{0.1in}
\begin{fileverb}
#!/bin/bash
\end{fileverb}

\vspace{0.1in}
\noindent
This is the first line of all Bash scripts and indicates where Bash is
located on your system. The location \var{/bin/bash} is a nearly
universal location across GNU/Linux and MacOS systems.

%====================================================================
\section*{The Duload Toolbox Scheme}
%====================================================================

The Duload Toolbox is built around the usage scenario depicted below,
designed to support a user working on a local computer with far less
storage capacity than required to have a local copy of a group's entire
project files:

\vspace{0.1in}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.43\linewidth]{figs_help/archiving.png}
  \caption{{\bf The Server and Local Machine}: Typically a user has far
    less memory on a local machine, and typically may want all or most
    low-res versions of files, but only a small subset of the high-res
    versions}
\label{fig_duload_usage}
\end{figure}

\noindent
In the Duload scheme, the string \var{"raw\_"} is used to begin
either a file name, or directory name, to indicate a high-resolution
version.  As an example, a folder may contain drone video files shot
at 4K \var{(3840x2160)} resolution, and the same files at a lower
resolution \var{(1137x640)}. For example:


\vspace{0.15in}
\begin{consoleverb}
 $ ls -hl
 -rw-rw-rw-  1 joe  staff   1.2G Aug 14 13:59 RAW_DJI_0003.MOV   // ~30x as big!
 -rw-rw-rw-@ 1 joe  staff   3.9G Aug 14 14:50 RAW_DJI_0004.MOV
 -rw-r--r--@ 1 joe  staff    35M Aug 14 16:07 DJI_0003.mp4
 -rw-r--r--@ 1 joe  staff   135M Aug 14 16:16 DJI_0004.mp4
\end{consoleverb}
\vspace{0.15in}

\noindent
The user may need the high resolution videos if composing an important
project video. But perhaps only the low-resolution video is wanted
after the project video is made, and the higher resolution version can
"archived" by not including it on the local drive.


\vspace{0.1in}
\noindent
Similar but different file organization styles may be used. For
example, a user may choose not to rename any files, but simply put the
lower resolution files into different folders:

\vspace{0.15in}
\begin{consoleverb}
 $ ls project
 raw_vids/
    DJI_0003.MOV
    DJI_0004.MOV
 vids/
    DJI_0003.MOV
    DJI_0004.MOV
\end{consoleverb}
\vspace{0.15in}

\noindent
In the above example, the "raw" nature of the files are indicated in
the directory portion in the full path name
\var{raw\_vids/DJI\_0003.MOV}. A group's project folder may be
constructed with several directory layers and different file naming
conventions. In the Duload scheme, any file with the term
\var{"raw"} in its full path name is regarded as a raw {\em version}
of a lower resolution file elsewhere in the project folder. Since these
raw files may be dispersed in several locations and layers in a folder,
the \cmd{rrf.sh} utility helps hunt these files down for removal when
space is needed. Furthermore, if the goal is to free up a certain amount
of memory, the \cmd{rrf.sh} utility can accept this target memory size
as a command line argument and only remove enough raw files to meet
this goal.

%=====================================================================
\subsection*{No Such Thing as a Duload Folder or File}

A folder structure used with Duload utilities is the same as any other
folder or directory in a GNU/Linux or MacOS system. The {\em
  convention} indicating a raw file is merely a convenience for
pulling down files from the server using \cmd{dload.pl} and removing
files locally with \cmd{rrf.sh}.

\vspace{0.1in}
\noindent
The user has flexibility in how raw files are demarcated: as part of
the file name, part of the directory name, or any part of the full
path name. The tools treat the \var{"raw\_"} string as case
insensitive, i.e., the following files are all considered raw files:

\vspace{0.1in}
\begin{packed_itemize}
  \item \var{raw\_video.mp4}
  \item \var{RAW\_IMG\_0123.mov}
  \item \var{raw\_videos/IMG\_0123.mov}
  \item \var{files/raw\_logs/LOG\_GUS\_18\_8\_2017\_\_\_\_\_20\_24\_14.alog}
\end{packed_itemize}
\vspace{0.1in}

\noindent
The following files are not considered raw files:

\vspace{0.1in}
\begin{packed_itemize}
  \item \var{draw\_video.mp4}
  \item \var{IMG\_0123\_RAW.mov}
  \item \var{videos\_raw/IMG\_0123.mov}
\end{packed_itemize}

%=====================================================================
\subsection*{How Un-Raw Files are Made}

By default a file in a project may be considered "un-raw". The notion
of a raw file only comes into play when a lower-resolution or smaller
version of an original file is created. At that point the original file
is simply regarded as raw, and usually re-named as such. Or put into
a folder named as such.

\vspace{0.12in}
\noindent
Some ways to make lower-resolution files:

\vspace{0.1in}
\begin{packed_itemize}
\item Saving clip of a video, for example a critical 20 second clip of
  an otherwise 30 minute video. Quicktime on MacOS supports this.
\item Creating a lower resolution version of a video. The TotalVideoConverter
  app and \cmd{ffmpeg} utility can do this.
\item Creating a lower resolution version of a photo. The
  \cmd{convert} utility in the ImageMagick package can do this. The
  MacOS Preview utility can also do this and many other tools.
\item Creating a clip of a MOOS alog file. The \cmd{alogclip} tool.
\item Creating a thinner version of a MOOS alog file. The
  \cmd{aloggrep}, \cmd{alogrm} and \cmd{alogpare} tools.
\end{packed_itemize}

%=====================================================================
\subsection*{Using the rrf.sh Utility}

The simplest way to use \cmd{rrf.sh} is to invoke it with no arguments.
In this case it will look for all raw files in your current directory
and recursively in sub-directories. It will tell you which files will
be removed, and how much memory will be freed, and will prompt you
for confirmation:

\vspace{0.1in}
\begin{consoleverb}
 $ rff.sh
 The following files will be removed: 
 Remove: 101MB   ./raw_vids/jake-state-finals.mov
 Remove: 2642MB   ./raw_vids/july2117-livestream.mp4
 Remove: 4201MB   ./raw_vids/july2317-livestream.mp4
 Total Size removed: 6944MB  (7GB)
 Do you wish to remove these files [y/N]?
\end{consoleverb}
\vspace{0.1in}

\noindent
The "hit list" of files to be removed is not sorted by size or date,
but is in the same recursive alphabetical order that the \cmd{ls}
command would provide.

By default, if the user just hits \var{ENTER} at this point, no action
will be taken. So this command also provides a preview service to the
user of all files that would be removed. If you really want to do this
operation without being prompted, use the \var{--force} option, at
your own risk.

%=====================================================================
\subsection*{Setting a Target Amount of Memory to Free}

If a user is interested in the \cmd{rrf.sh} utility, it is likely because
the file system is near budget or capacity before making room for new
files. So typically the user has a target amount of memory in mind, and
is not interested in removing all raw files. The \cmd{rrf.sh} utility
accepts a numerical value on the command line which indicates the number
of {\em gigabytes} to be removed. Using the previous example:

\vspace{0.1in}
\begin{consoleverb}
 $ rff.sh 2
 The following files will be removed: 
 Remove: 101MB   ./raw_vids/jake-state-finals.mov
 Remove: 2642MB   ./raw_vids/july2117-livestream.mp4
 Total Size removed: 2743MB  (3GB)
 Do you wish to remove these files [y/N]?
\end{consoleverb}
\vspace{0.1in}

\noindent
Notice that the third file from the previous example, is no longer on
the hit list, since removing the first two files alone would free up
the requested 2GB of memory. Note that \cmd{rrf.sh} will continue
removing files until the target has been {\em met or exceeded}. For
example, if the target in the above example were 1GB, the same two
files would have been removed. The utility is not smart enough to
realize that removing the second file alone would have sufficed.

%=====================================================================
\subsection*{Setting File Size Threshold - Skip the Small Fish}

By default, the \cmd{rrf.sh} utility will not remove files unless they
are above 10MB. This heuristic aimed at not removing raw photos, which
typically consume much less memory than video or mission log
files. This can be overridden on the command line with the
\var{--thresh=N} option.  It can used for example to target only
videos above 4GB. From our previous example:

\vspace{0.1in}
\begin{consoleverb}
 $ rff.sh
 The following files will be removed: 
 Remove: 4201MB   ./raw_vids/july2317-livestream.mp4
 Total Size removed: 4201MB  (4GB)
 Do you wish to remove these files [y/N]?
\end{consoleverb}
\vspace{0.1in}

\noindent
Note the two smaller files from the previous example are omitted from the
hit list, and only the file bigger than 4GB is removed.

%=====================================================================
\subsection*{Command Line Help}

As with most command line tools, \cmd{rrf.sh --help} should refresh you
on the important stuff..

\end{document}



</pre></body></html>