<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[onecolumn,letterpaper,11pt]{article}

%===================================================================
\input{common_latex/common_latex_packages}
\input{common_latex/common_latex_definitions}
\input{common_latex/common_latex_help_format}
%===================================================================

\begin{document}

\begin{labcover}
\labtitle{uSpeechRec: Julius Speech Recognition}
\vspace{0.1in}
\labdate{Spring 2019}
\vspace{0.1in}
\labauth{Michael Novitzky, novitzky@mit.edu}
\labauth{Department of Mechanical Engineering}
\labauth{MIT, Cambridge MA 02139}
\labtoc{small}
\end{labcover}

\wikidoc{file=PavLab.USpeechRec}
\wikidoc{bugz=uSpeechRec}
\wikidoc{auth=novitzky@mit.edu}

%=================================================================
\section{uSpeechRec: Julius Speech Recognition }
%=================================================================
\label{sec_uspeech}
\index{uSpeechRec}

The \app{uSpeechRec} application, now in version 3.0, is a module for
using the Open-Source Large Vocabulary CSR Engine Julius.  It is an
interface between Julius and the middleware MOOS.  By using
\app{uSpeechRec} along with a dialogmanager, such as
\app{uDialogManager}, a person can use a microphone and simple speech
to interact with other MOOS applications.

\noindent
Key parts needed for \app{uSpeechRec} are:

\begin{packed_enumerate}[3]
\item {\em Julius Configuration File}: A configuration file, such as
  Alpha.jconf, specifies important options for running Julius.  Many
  modifications can be made through the configuration file.  By
  default, the configuration file specifies the use of a microphone
  and is setup for the VoxForge configuration including their acoustic
  model.

\item {\em Acoustic Model Files}: An acoustic model represents the
  relationship between an audio signal and the phonemes that make up
  speech.  Distributed with \app{uSpeechRec} is an English version of
  the acoustic model provided by VoxForge.net.

\item {\em Grammar}: A grammar specification in speech recognition is
  a set of word patterns and informs the speech recognition system
  what to expect a human to say.  A sample grammar from VoxForge.net
  has been modified for the purposes of \app{uSpeechRec}.
  Explanations on how to modify and create new grammars are described
  below.

\end{packed_enumerate}

%====================================================================
\section{Building uSpeechRec}
%======================================================================

Prior to building \app{uSpeechRec} you must install some dependencies.

%======================================================================
\subsection{MAC OS X}

We recommend using MacPorts to install dependencies on MAC OS X. You
can add the packages for Julius and portaudio:

\vspace{0.15in}
\begin{fileverb} 
sudo port install julius portaudio
\end{fileverb}
\vspace{0.15in}

\noindent
You MUST then add to either your .bash\_profile, .bashrc, or .profile
the corresponding library path:

\vspace{0.15in}
\begin{fileverb} 
 export LIBRARY_PATH=/opt/local/lib
\end{fileverb}
\vspace{0.15in}

\noindent
It is necessary because unfortunately MacPorts is not properly adding
the library path as of this writing.

%======================================================================
\subsection{Ubuntu}

In order to install the Ubuntu dependencies simply enter into the command line:


\vspace{0.15in}
\begin{fileverb} 
 $ sudo apt-get install julius julius-dev libportaudio-dev libpulse-dev libasound2-dev
\end{fileverb}
\vspace{0.15in}

%======================================================================
\subsection{Build}

To build on Linux and Apple platforms, execute the build script within this
directory:

\vspace{0.15in}
\begin{fileverb} 
   $ ./build.sh
\end{fileverb}
\vspace{0.15in}

To build without using the supplied script, execute the following commands
within this directory:

\vspace{0.15in}
\begin{fileverb} 
   $ mkdir -p build
   $ cd build
   $ cmake ../
   $ make
   $ cd ..
\end{fileverb}
\vspace{0.15in}

%======================================================================
\section{Using uSpeechRec}
%======================================================================

Typical use of \app{uSpeechRec} has it situated in a community in
which a human will interact with it using speech.  In addition to
\app{uSpeechRec} another application in the same community must
interpret the sentences from \app{uSpeechRec} and deterimine how it
should interact with the system.

%======================================================================
\subsection{Typical Module Topology}

The typical module topology is shown in Figure \ref{fig_aq_topology}
below. The \app{uSpeechRec} is situated in a community in which speech
will be used as a form of interaction.  It is typically run alongside
\app{uDialogManager} and \app{iSay} for an interactive experience.
The \app{uSpeechRec} application subscribes to the variables
\mvar{SPEECH\_ACTIVE} and \mvar{SPEECH\_PAUSE} which perform the same function
of muting the microphone input which mimics a paused state.
The \app{uSpeechRec} publishes the most likely sentence to
\mvar{SPEECH\_RECOGNITION\_SENTENCE} along with the confidence scores to
\mvar{SPEECH\_RECOGNITION\_SCORE}.  If any errors in speech recognition occur,
\app{uSpeechRec} publishes to \mvar{SPEECH\_RECOGNITION\_ERROR}.

\vspace{0.1in}
\begin{figure}[H]
  \centering 
  \includegraphics[width=0.7\textwidth]{figures/aq_topology.png}
  \caption{{\bf Typical uSpeechRec Topology:} This module runs in any
    community in which one would like to use Speech Recognition.  It
    is typically used with the applications \app{uDialogManager} and
    \app{iSay} for an interactive experience.}
\label{fig_aq_topology}
\end{figure}

%======================================================================
\subsection{Pausing uSpeechRec}
%======================================================================

There may be situations in which you want to pause \app{uSpeechRec}.
Pausing is accomplished by posting \var{TRUE} to the MOOSDB variable
\mvar{SPEECH\_PAUSE}.  \app{uSpeechRec} will remain paused until \var{FALSE} is
posted \mvar{SPEECH\_PAUSE}.  The status of \mvar{SPEECH\_PAUSE} can be seen through
AppCasting. Alternatively, by posting the opposite variables to
\mvar{SPEECH\_ACTIVE} will gain the same effect as pausing.


%======================================================================
\subsection{Starting uSpeechRec in a Paused State}
%======================================================================
\label{sec_start_paused}
There may be situations in which you want to have \app{uSpeechRec}
start in a paused state. Starting \app{uSpeechRec} in a paused state
is accomplished by inserting StartState = Paused in the .moos file for
\app{uSpeechRec}.  In order to have recognition start then a posting of
\var{FALSE} needs to be posted to the MOOSDB variable
\mvar{SPEECH\_PAUSE}.

%======================================================================
\subsection{Interpreting the Outputs}
The most likely output speech recognition sentence by Julius is published to the
\mvar{SPEECH\_RECOGNITION\_SENTENCE}.  However, just because it is the most
likely sentence based on the acoustic model, vocabulary, and grammar files, does
not mean that is confident that is the correct sentence.  To get further
confidence, starting with \app{uSpeechRec} version 3.0, it is best to look at
the \mvar{SPEECH\_RECOGNITION\_SCORE}.  Here is an example
\mvar{SPEECH\_RECOGNITION\_SCORE} 

sentence= BLUE THREE ATTACK, confidencescores=1:0.999692:0.912241:0.73853:1, score1= -12884.4

The sentence section is a repetition of the most likely recognized sentence.
The confidencescores section includes the confidence for each word separated by ':'.  The first
and last numbers reflect the starting silence and ending silence - which is
typically 1.  The numbers in between are the confidence, (0.0-1.0], that Julius
has that the word is the correct word. This output can be leveraged by
uDialogManager to reject a sentence based on a word confidence threshold.


%======================================================================
\subsection{The SPEECH\_RECOGNITION\_SENTENCE Format}
\label{sec_speech_rec_sentence}

The available set of sentences that uSpeechRec will recognize is tied
to the \var{.grammar} file and the \var{.voca} file.  The
\var{.grammar} file describes the structure of the available
sentences.  The \var{.voca} file describes the vocabulary that
comprise the sentences.

\noindent
Let's take a simple \var{.grammar} file:


\vspace{0.15in}
\begin{fileverb} 
S : NS_B SENT NS_E

SENT: NAME COMMAND
SENT: ACK
\end{fileverb}
\vspace{0.15in}
 
\noindent 
Here we see a simple grammar specification starting with the first
line:

\vspace{0.15in}
\begin{fileverb} 
S : NS_B SENT NS_E
\end{fileverb}
\vspace{0.15in}

\noindent 
Where S is the root and it is defined with \var{NS\_B} which
corresponds to silence with a \var{SENT} or sentence and terminated
with \var{NS\_E} which is also silence.  In the following line
\var{SENT} is defined:

\vspace{0.15in}
\begin{fileverb} 
SENT: NAME COMMAND
\end{fileverb}
\vspace{0.15in}

\noindent

A possible definition for \var{SENT} or sentence is created with the
terms \var{NAME} followed by \var{COMMAND}.  A second definition for
\var{SENT} is created on the following line:

\vspace{0.15in}
\begin{fileverb} 
SENT: ACK
\end{fileverb}
\vspace{0.15in}

\noindent 
in which \var{SENT} or sentence is defined as \var{ACK}.

The words such as \var{NAME}, \var{COMMAND}, and \var{ACK} are defined
in the \var{.voca} file.  The \var{.voca} file uses the syntax \%
followed by grammar construct such as \var{NAME}.  Below \var{NAME}
are the text such as \var{ARNOLD} followed by the phonemes such as
\var{aa r n ah l d}.  See the following example:
 
\vspace{0.15in}
\begin{fileverb} 
% NS_B
&lt;s&gt;        sil

% NS_E
&lt;/s&gt;        sil

% NAME
ARNOLD     aa r n ah l d
BETTY      b eh t iy
CHARLIE    ch aa r l iy
TEAM       t iy m

%COMMAND
RETURN     r ih t er n
DEPLOY     d ih p l oy
FOLLOW     f aa l ow
STATION    s t ey sh ah n

%ACK
YES        y eh s
NO         n ow
\end{fileverb}
\vspace{0.15in}

\noindent 
The configuration file for Julius has the \var{.jconf} extension.  As
an example we include the configuration block suggested by
VoxForge.org:

\vspace{0.15in}
\begin{fileverb} 
# VoxForge configurations:
-input mic    			# live microphone
-dfa grammar/alpha.dfa
-v grammar/alpha.dict
-h acoustic_model_files/hmmdefs
-hlist acoustic_model_files/tiedlist
-spmodel "sp"		# HMM model name
-multipath
-gprune safe
-iwcd1 max
-iwsppenalty -70.0	# transition penalty for the appended sp models
-smpFreq 16000		# sampling rate (Hz)
-iwsp			# append a skippable sp model at all word ends
-penalty1 5.0		
-penalty2 20.0	
-b2 200                 # beam width on 2nd pass (#words)
-sb 200.0		# score beam envelope threshold
-n 1
# !!!!!!
\end{fileverb}
\vspace{0.15in}

\label{sec_alpha_jconf}

\noindent 
The key parameters describe where the grammar files are found and the
acoustic model can be found.  In this example the two grammar file
parameters are:

\vspace{0.15in}
\begin{fileverb} 
-dfa grammar/alpha.dfa
-v grammar/alpha.dict
\end{fileverb}
\vspace{0.15in}

\noindent 
The following parameters describe where the acoutic files are to be
found:

\vspace{0.15in}
\begin{fileverb} 
-h acoustic_model_files/hmmdefs
-hlist acoustic_model_files/tiedlist
\end{fileverb}
\vspace{0.15in}

\noindent
The rest of the parameters in the VoxForge.org \var{.jconf} file
describe the model parameters which are beyond the scope of this
document.

%======================================================================
\section{Adding New Words To Your Grammar}
%======================================================================

Because we are using the VoxForge.org dictionary, we are limited to
the words and phones contained within.  This was the dictionary used
to train the acoustic model.  However, if there are words within the
dictionary you wish to use it is relatively a simple process to add
them to the vocabulary list. Let's say you would like to add the word
\var{STATUS} to your list of commands.  That way you can ping a robot
to get it's status by saying \var{ARNOLD STATUS}.  First, we look into
the grammar folder and search through the file \var{VoxForgeDict.txt}
for the word \var{STATUS}. It seems that we have found two entries
with the first being:


\vspace{0.15in}
\begin{fileverb} 
STATUS          [STATUS]        s t ae t ah s 
\end{fileverb}
\vspace{0.15in}

\noindent
In this example we are running \var{alpha.voca} as our vocabulary
file.  We will add just the english text \var{STATUS} and phones
\var{s t ae t ah s} to the last entry under \var{\%COMMAND}.


\vspace{0.15in}
\begin{fileverb} 
% COMMAND
RETURN     r ih t er n
DEPLOY     d ih p l oy
FOLLOW     f aa l ow
STATION    s t ey sh ah n
STATUS     s t ae t ah s 
\end{fileverb}
\vspace{0.15in}

\noindent
But we are not finished just yet.  We must recompile the Julius
grammar called \var{alpha} using the Perl script \var{mkdfa.pl}.

\vspace{0.15in}
\begin{consoleverb} 
 $ mkdfa.pl alpha
\end{consoleverb}
\vspace{0.15in}

\noindent
If this worked properly, \var{mkdfa.pl} converted the \var{.grammar}
and \var{.voca} file into Julius specific formats of \var{.dfa} and
\var{.dict} files.  Now, when we run the mission we can use the phrase
\var{NAME COMMAND} with \var{ARNOLD STATUS} as an example.
  
%======================================================================
\section{Modifying Grammar}
%======================================================================

Let's say that we have exhausted the original sentence structure in
the \var{alpha.grammar} file.  For example, we really enjoyed using
\var{ARNOLD FOLLOW} to mean that the robot \var{ARNOLD} should
\var{FOLLOW} me. However, this is a limited vocabulary because we
would like to specify an object to some of our commands.  For example,
I would like for \var{ARNOLD} to \var{FOLLOW} another robot such as
\var{BETTY}.  At the moment, our \var{alpha.grammar} does not allow
for such a sentence.  Let's add this powerful grammar to it!  Our
original \var{alpha.grammar} specifies two sentence formats.


\vspace{0.15in}
\begin{fileverb} 
S : NS_B SENT NS_E

SENT: NAME COMMAND
SENT: ACK
\end{fileverb}
\vspace{0.15in}
 
\noindent
Let's add a third sentence format by inserting the following line:

\vspace{0.15in}
\begin{fileverb} 
 SENT: NAME COMMAND NAME
\end{fileverb}
\vspace{0.15in}
 
\noindent
This sentence format allows us to specify any name in our \var{.voca}
file followed by any \var{COMMAND} with the object \var{NAME}.  Our
final \var{.grammar} file will look like:

\vspace{0.15in}
\begin{fileverb} 
S : NS_B SENT NS_E

SENT: NAME COMMAND NAME
SENT: NAME COMMAND
SENT: ACK
\end{fileverb}
\vspace{0.15in}

\noindent
We will need to convert this updated \var{.grammar} file into
something Julius can use.

\vspace{0.15in}
\begin{consoleverb} 
 $ mkdfa.pl alpha
\end{consoleverb}
\vspace{0.15in}
 
\noindent
The Perl script \var{mkdfa.pl} has taken a \var{.grammar} file and a
\var{.voca} file and converted them into Julius formats of \var{.dfa}
and \var{.dict}. Now when running \app{uSpeechRec}, a user can use the
additional sentence \var{NAME COMMAND NAME} such as \var{ARNOLD FOLLOW
  BETTY}.  It is up to the user to determine how this new sentence
will be used in a second app such as \app{uDialogManager}.

%========================================================================
\section{Configuration Parameters of uSpeechRec}
\index{uSpeechRec!Configuration Parameters}
\index{Configuration Parameters!uSpeechRec}

The following parameter is defined for \app{uSpeechRec}. A more
detailed description is provided in other parts of this
section. Parameters having default values are indicated so.

\vspace{0.15in}
\begin{listing}{Configuration Parameters for uSpeechRec}
\label{lst_uSpeechRec_params}
  \renewcommand{\arraystretch}{1.1}
  \begin{longtable}{rp{0.8\textwidth}} 
    
    \pvar{JuliusConf}: &amp; Names the file for the Julius
    configuration. For example \var{"JuliusConf = Alpha.jconf"}. See
    Section \ref{sec_alpha_jconf}. \\

    \pvar{StartState}: &amp; Decides the Paused state that \app{uSpeechRec} will
    start in. Options are 'Active' or 'Paused', see Section \ref{sec_start_paused}. 
    
  \end{longtable}
\end{listing}
  
%======================================================================
\subsection{An Example MOOS Configuration Block}

To see an example MOOS configuration block, enter the following from
the command-line:

\vspace{0.1in}
\begin{consoleverb}
 $ uSpeechRec --example or -e
\end{consoleverb}
\vspace{0.1in}

\noindent
This will show the output shown in Listing \ref{lst_uSpeechRec_econfig} below.

\vspace{0.15in}
\begin{listing}{Example configuration of the uSpeechRec application}
\label{lst_uSpeechRec_econfig}
\footnotesize
\begin{verbatim}
   1  =============================================================== 
   2  uSpeechRec Example MOOS Configuration                     
   3  =============================================================== 
   4                                                                  
   5  ProcessConfig = uSpeechRec                                
   6  {                                                               
   7    AppTick   = 4                                                 
   8    CommsTick = 4                                                 
   9                                                                  
  10    JuliusConf = Alpha.jconf
  11    //StartState default is 'Active'
  12    //Can be set to 'Paused'.  Must then be unpaused
  13    //by posting to the MOOSDB SPEECH_PAUSE=FALSE
  14    StartState = Paused 
  15  }                                                          
\end{verbatim}
\normalsize
\end{listing}

%================================================================
\section{Publications and Subscriptions for uSpeechRec}
%======================================================================
\index{uSpeechRec!Publications and Subscriptions}
\index{Publications and Subscriptions!uSpeechRec}
\label{sec_uSpeechRec_interface}

The interface for \app{uSpeechRec}, in terms of publications and
subscriptions, is described below. This same information may also be
obtained from the terminal with:

\begin{consoleverb}
 $ uSpeechRec --interface or -i
\end{consoleverb}

\subsection{Variables Published by uSpeechRec}
\label{sec_uSpeechRec_pubs}

\begin{packed_itemize}[2]
\item \mvar{APPCAST}: Contains an appcast report identical to the
  terminal output. Appcasts are posted only after an appcast request
  is received from an appcast viewing utility.
\item \mvar{SPEECH\_RECOGNITION\_SENTENCE}: The most likely sentence heard by 
the Julius Speech Recognition Engine. Section \ref{sec_speech_rec_sentence}.
\item \mvar{SPEECH\_RECOGNITION\_SCORE}: The most likely sentence heard by 
the Julius Speech Recognition Engine and the confidence scores in each word. Section \ref{sec_speech_rec_sentence}.
\item \mvar{SPEECH\_RECOGNITION\_ERROR}: If Julius Speech Recognition Engine
  does not produce a result then an error is output here..
\end{packed_itemize}

%======================================================================
\subsection{Variables Subscribed for by uSpeechRec}
\label{sec_uSpeechRec_subs}

The \app{uSpeechRec} application will subscribe for the following
four MOOS variables:

\begin{packed_itemize}
\item \mvar{APPCAST\_REQ}: A request to generate and post a new
  apppcast report, with reporting criteria, and expiration.

\item \mvar{SPEECH\_PAUSE}: A value of \var{TRUE} will pause 
   \app{uSpeechRec} while a value of \var{FALSE} will 
   unpause/start \app{uSpeechRec}.

 \item \mvar{SPEECH\_ACTIVE}: Used as an alternative to \mvar{SPEECH\_PAUSE}. A value of \var{TRUE} will unpause 
   \app{uSpeechRec} while a value of \var{FALSE} will 
   pause \app{uSpeechRec}.
   
\end{packed_itemize}

%======================================================================
\subsection{Command Line Usage of uSpeechRec}
\label{sec_uSpeechRec_cmdline}
\index{Command Line Usage!uSpeechRec}
\index{uSpeechRec!Command Line Usage}

The \app{uSpeechRec} application is typically launched as a part
of a batch of processes by \cmd{pAntler}, but may also be launched from the
command line by the user.  To see command-line options enter the
following from the command-line:

\begin{consoleverb}
 $ uSpeechRec --help or -h
\end{consoleverb}

\noindent
 This will show the output shown in Listing \ref{lst_uSpeechRec_usage} below.

\vspace{0.25in}
\begin{listing}{Command line usage for uSpeechRec}
\label{lst_uSpeechRec_usage}
  
\footnotesize
\begin{verbatim}
   1  ==========================================================
   2  Usage: uSpeechRec file.moos [OPTIONS]               
   3  ==========================================================
   4                                                            
   5  Options:                                                  
   6    --alias=&lt;ProcessName&gt;                                
   7        Launch uFldHazardMetric with the given process name. 
   8    --example, -e                                           
   9        Display example MOOS configuration block.           
  10    --help, -h                                              
  11        Display this help message.                          
  12    --interface, -i                                         
  13        Display MOOS publications and subscriptions.        
  14    --version,-v                                            
  15        Display release version of uSpeechRec.        
\end{verbatim}
\normalsize
\end{listing}

%======================================================================
\section{Terminal and AppCast Output}
%======================================================================
\label{sec_uSpeechRec_appcast}

\vspace{0.1in}
\begin{listing}{Example uSpeechRec console output}
  \label{lst_uSpeechRec_appcast}

\footnotesize
\begin{verbatim}
   1  ===================================================================
   2  uSpeechRec mokai                                           0/0(682)
   3  ===================================================================
   4  ====================================================
   5  JuliusConf = Alpha.jconf
   6  Recognizer Paused
   7  ===================================================================
   8  Most Recent Events (2)
   9  ===================================================================     
   10  sentence: BLUE THREE ATTACK, confidencescores: 1 0.999692 0.912241 0.73853 1, score1: -12884.4
   11  sentence: NO, confidencescores: 1 0.368944 1, score1: -10499.8  

\end{verbatim}
\normalsize
\end{listing}

\noindent 
Line 2 includes the name or alias that uSpeechRec is using followed by the community
in which it resides.  
The final set of numbers on line 2 include the number of appcasts that have been published.
Line 5 includes the \var{.jconf} file being used by the Julius Speech Recognition Engine for 
configuration. 
Line 6 presents whether the speech recognition provided by Julius is 'Paused' or 'Active'.
During normal operations Lines 10-11 provide a history of the latest
 sentences recognized by the Julius Open-Source Recognition Engine.  
Immediately following the most likely recognized sentence is the negative log-likelihood 
provided by Julius.
If there is a configuration or runtime warning then Line 5 would be replaced
with \var{.jconf} file issues or the possible situation when Julius could 
not be initialized. 

\end{document}


</pre></body></html>