07-runtime.tex

\section{Run-time Control of Fault Injection}
\label{sec:runtime}

There are two options available to control the \textit{\ac{FIJI}} logic added to the instrumented design in the target FPGA device at run time.
The \textit{\ac{FIJIEE}} tool is a command-line tool which facilitates downloading pre-defined or random test patterns but can also be controlled interactively.
The \textit{\ac{FIJIEE} GUI} tool provides a graphical user interface to roughly the same functionality.
Both communicate with \textit{\ac{FIJI}} via serial connection (UART) to download test patterns and read back status information.

Both \textit{\ac{FIJIEE}} programs work with \textit{\ac{FIJI} Tests} files, which can
contain (1) overall test execution parameters such as the serial device
to be used for communication with the fault injection logic, (2) 
a succession of test patterns for sequenced tests along with test
execution information, and (3) test execution information for random
tests such as fault probabilities or minimum and maximum timer values.

\Cref{sec:execution_flow} explains how the \ac{FIJI} hardware and the \ac{FIJIEE} on the host PC communicate to coordinate the execution of tests.
A description of the \ac{FIJIEE} command-line tool and GUI follow in \Cref{sec:ee-cli} and \Cref{sec:ee-gui} respectively.
Specific features that extend the run-time functionality are detailed in the trailing sections.

\subsection{Execution Flow}
\label{sec:execution_flow}

\begin{figure}[ht]
\centering
\includegraphics[width=0.85\linewidth]{img/Pattern.pdf}
\caption{Fault Sequencing}
\label{fig:seq}
\end{figure}


In \Cref{fig:seq} an extract of a longer execution of a \ac{FIJI} system is depicted showing 
some of the possible sequences of events.

First of all, the \ac{FIC} must acquire a valid configuration (as specified in \hyperref[TRM:sec:h2f]{the \ac*{TRM}}).
If there are any reception errors (frame error, wrong 
ID, or CRC mismatch) the \ac{FIC} nevertheless continues reading in the appropriate 
amount of frames/bytes for a configuration. Only after the complete reception 
of the expected number of bytes the \ac{FIC} reports the result back to the host. To 
that end a \textit{CONF\_DONE} message is sent, possibly with some set error bits (see 
\hyperref[TRM:sec:f2h]{the \ac*{TRM}}).

Upon the successful reception of a configuration, the DUT is reset by the \ac{FIC} 
if requested. The \ac{FIC} then optionally waits for a trigger signal (again 
depending on a configuration setting) and then starts the next phase.
NB: In \Cref{fig:seq} the 
second and partially visible third execution cycle do not use any triggers and 
thus have no \textit{WAIT} phases at all.

In the \textit{COUNT} phase the timer counts down duration $t_1$. When the timer expires, 
the \ac{FIC} instructs the \acp{FIU} to apply fault pattern 1 and thus possibly enables 
the forwarding of faulty signals by the \acp{FIU}. At this point the \ac{FIC} reports 
back to the host that transmitting a new configuration is now possible with a 
\textit{READY} message. Also, it loads the timer with duration $t_2$. When the timer 
expires, the \ac{FIC} instructs the \acp{FIU} to apply fault pattern 2. This pattern 
stays active until the duration $t_1$ of the next configuration has been counted 
down.

In other words $t_1$ determines exactly the time between reception of a new configuration or optionally the arrival of a trigger and the start of the first fault.
$t_2$ on the other hand specifies how long this first fault should be active (before fault~2 becomes active).

In the ideal case the host sends a new configuration while duration $t_2$ of the 
previous configuration is still in progress. This allows to apply successive 
configurations or even fault patterns themselves without a gap. If the new 
configuration does not arrive in time, the \ac{FIC} notifies the host by sending an
\textit{UNDERRUN} message. It then waits for a new configuration to be downloaded.


\subsection[FIJIEE Command-line Interface]{\acs{FIJIEE} Command-line Interface}
\label{sec:ee-cli}

The \textit{\ac{FIJIEE}} may be invoked as follows:

\texttt{perl fiji\_ee.pl [OPTIONS]}

The following command-line parameters are required regardless of operation mode:

\begin{itemize}
    \item \texttt{-m, {-}{-}mode=<mode>}
          
    The download mode to use:
    \begin{itemize}
        \item[] \texttt{manual}: Prompt for fault patterns and timer values
        \item[] \texttt{auto}:   Download tests in the \textit{\ac{FIJI} Tests} file
        \item[] \texttt{random}: Download randomly generated tests
    \end{itemize}

    \item \texttt{-s, {-}{-}settings=<filename>}

    Specifies the \textit{\ac{FIJI} Settings} file to use.

    \item \texttt{-t, {-}{-}tests=<filename>}

    Specifies the \textit{\ac{FIJI} Tests} file to use.
\end{itemize}

The script also accepts the following \emph{optional} command-line arguments:

\begin{itemize}
    \item \texttt{-p, {-}{-}port=<port>}

        The serial port to use.
        Overrides the value configured in the \textit{\ac{FIJI} Settings} file.

    \item \texttt{-c, {-}{-}completion=<filename>}

        Path to the completion script (cf.\ \Cref{sec:completion_script}).
        Overrides the value configured in the \textit{\ac{FIJI} Tests} file.

    \item \texttt{-i, {-}{-}interactive}

        Enables user interaction (default in \texttt{manual} mode).
        \textit{\ac{FIJIEE}} is supposed to be run non-interactively most of the time (e.g., when called by test scripts).
        However, if this option is given \textit{\ac{FIJIEE}} offers the user some options which require user input (e.g., to save test files for failures etc).

    \item \texttt{-v, {-}{-}verbose}

        Increase verbosity on the terminal. Each occurence of \texttt{-v} or
        \texttt{{-}{-}verbose} further increases verbosity.

    \item \texttt{-h, {-}{-}help}

        Displays usage information on the terminal and exits
\end{itemize}

In \textit{Random} mode, the following options can be used to overrule
the values defined in the supplied \textit{\ac{FIJI} Tests} file:

\begin{itemize}
    \item \texttt{-a, {-}{-}timer-1-min=<duration>}
    
    Minimum value for duration $t_1$

    \item \texttt{-b, {-}{-}timer-2-min=<duration>}
    
    Minimum value for duration $t_2$

    \item \texttt{-A, {-}{-}timer-1-max=<duration>}

    Maximum value for duration $t_1$

    \item \texttt{-B, {-}{-}timer-2-max=<duration>}

    Maximum value for duration $t_2$

    \item \texttt{-0, {-}{-}prob-s0=<probability>}

    Probability of a stuck-at-0 fault on any net

    \item \texttt{-1, {-}{-}prob-s1=<probability>}

    Probability of a stuck-at-1 fault on any net

    \item \texttt{-d, {-}{-}prob-dly=<probability>}

    Probability of a delay fault on any net

    \item \texttt{-f, {-}{-}prob-seu=<probability>}

    Probability of an SEU fault on any net

    \item \texttt{-o, {-}{-}prob-so=<probability>}
    
    Probability of a stuck-open fault on any net
\end{itemize}

The entered durations must fit into the timer width specified in the
\textit{\ac{FIJI} Settings} file, with the maximum duration being greater
than or equal to the minimum duration.

The sum of the fault probabilities must not exceed $1.0$. The difference
between this sum and $1.0$ determine the probability of fault-free
operation.

When choosing the \texttt{manual} mode, \textit{\ac{FIJIEE}} prompts
the user for fault configurations before they are downloaded as shown
in \Cref{lst:manualprompt}.

To choosen between the various fault models one has to enter its index.
The mapping between these fault indices and the respective fault models is shown in \Cref{tab:faultmap}.

\begin{figure}
\begin{lstlisting}[caption=Interactive configuration session,label=lst:manualprompt,style=plain]
Enter configuration for FIU #0 in t2 (default: 0x7): 0
Enter configuration for FIU #0 after t2 (default: 0x7): 1
Enter configuration for FIU #1 in t2 (default: 0x7): 2
Enter configuration for FIU #1 after t2 (default: 0x7): 3
Enter configuration for FIU #2 in t2 (default: 0x7): 4
Enter configuration for FIU #2 after t2 (default: 0x7): 5
Enter duration t1 (default: 0x0): 100
Enter duration t2 (default: 0x0): 1000
Enable trigger (default: 0)? 1
Use external/not internal trigger (default: 0)? 0
Enable reset (default: 0)? 0
\end{lstlisting}
\end{figure}

\begin{table}[!hb]
    \caption{Mapping between fault indices and models}
    \label{tab:faultmap}
    \centering
    \begin{tabular}{ll}
        \toprule
        Index & Fault Model \\
        \midrule
        0 & No fault \\
        1 & Stuck at 0 \\
        2 & Stuck at 1 \\
        3 & Delay \\
        4 & Single Event Upset \\
        5 & Stuck open \\
        6 & Reserved - No fault \\
        7 & Reserved - No fault \\
        \bottomrule
    \end{tabular}
\end{table}

Following the fault configurations for the different \ac{FIU}s in both phases 
the time values for both duration have to be entered in clock cycles.

Afterwards, the trigger mode must be selected, where
\texttt{1} maps to enabling the trigger facility. If the user enters a \texttt{1},
\textit{\ac{FIJIEE}} asks if the internal trigger (\texttt{0}) or the
external trigger (\texttt{1}) shall be used.

Finally, the user has to enter if the fault injection logic shall apply
a reset to the \ac{DUT} before applying this fault injection pattern. 

Informations other than the fault patterns (e.g., the serial port,
baud rate, or \texttt{HOLD\_ON\_FAULT\_DETECT} information) are read from
the \textit{\ac{FIJI} Settings} and \textit{\ac{FIJI} Tests} files.


\subsection[FIJIEE GUI]{\acs{FIJIEE} GUI}
\label{sec:ee-gui}

Apart from the functionality explained above the \acs{FIJIEE} GUI additionally provides the possibility to edit \textit{\ac{FIJI} Tests} files in a guided manner, where entered values are checked for validity and sanity.

Although the \textit{\ac{FIJIEE}} GUI is a graphical tool, it accepts the
following \emph{optional} command-line arguments:

\begin{itemize}
    \item \texttt{-s, {-}{-}settings=<filename>}

        Specifies a \textit{\ac{FIJI} Settings} file to load at startup.

    \item \texttt{-t, {-}{-}tests=<filename>}

        Specifies a \textit{\ac{FIJI} Tests} file to load at startup.
    
    \item \texttt{-v, {-}{-}verbose}

       Increase verbosity on the terminal. Each occurence of \texttt{-v} or
       \texttt{{-}{-}verbose} further increases verbosity.

    \item \texttt{-h, {-}{-}help}

        Displays usage information on the terminal and exits
\end{itemize}

The main elements of the \textit{\ac{FIJIEE} GUI} can be seen in \Cref{fig:download_gui}:
\begin{enumerate}[(1)]
    \item The \textit{Control Area} is used for loading and saving \textit{\ac{FIJI} Tests} files, as well as
          for exporting test runs to \textit{\ac{FIJI} Tests} files or templates for \ac{RTL} and gate-level simulation.
          Here the user can also select the UART device to be used for communication with the \ac{FIJI} logic.
    \item The \textit{Tab Area} switches between test modes.
    \item In the \textit{Main Area}, the tests in the selected mode
          can be configured.
    \item The \textit{Logging Window} shows the progress and outcome of operations.
          Warnings and error messages are color-coded in orange and red, respectively.
    \item Small green or red panels show the most recent status reported
          back by the \ac{FIJI} logic in the \textit{\ac{FIC} Status Panel}:
          \begin{itemize}
            \item \textit{U}: UART framing error
            \item \textit{I}: Design ID mismatch (between the hardware and the \ac{FIJI} configuration file)
            \item \textit{C}: CRC error
            \item \textit{1}: Fault detect signal 1
            \item \textit{2}: Fault detect signal 2
          \end{itemize}
    \item The \textit{Injection Status Panel} shows the running time of the current test on the right and type of the last received status message on the left.
          \begin{itemize}
            \item \textit{CONF\_DONE}: reception of the configuration succeeded
            \item \textit{READY}:      $t_1$ of the test has passed and \ac{FIJI} is ready to receive another configuration
            \item \textit{UNDERRUN}:   $t_2$ ended before a new test configuration was received (thus no exact timing can be maintained)
          \end{itemize}
    \item The buttons in the \textit{Button Panel} allow the user to control the test execution:
          \begin{itemize}
            \item \textit{Clear Log}: Clear the contents of the log window.
            \item \textit{Update}:    When the \ac{FIC} is ready (after $t_2$), request
                                      an update of the fault detect signals.
            \item \textit{Start}:     Run the currently configured test.
            \item \textit{Stop}:      Abort the current execution of tests.
          \end{itemize}
\end{enumerate}

\begin{figure}[!htb]
\centering
\input{img/fiji_ee_sequence_screenshot_marked.tex}
\caption{\ac{FIJIEE} GUI: Sequence Mode}
\label{fig:download_gui}
\end{figure}

\subsubsection{Sequence}
\label{sec:sequence}

In \textit{Sequence} mode, the \textit{\ac{FIJIEE} GUI} provides the
possibility to edit a sequence of fault patterns to be downloaded one
after another. At the top of the \textit{Main Area} of the sequence mode tab
generic parameters for the test execution can be set: If the
sequence is to be repeated once completed, and after which pattern the repetition
should start again (this is particularly useful if the first pattern is
used to reset the \ac{DUT}), and under which circumstances to halt the test execution.
In addition to these configurable halting conditions, the test execution is always stopped if and when an ID or CRC error is encountered.

Additionally, a Perl script can be selected to be executed upon test completion, which is described in detail in \Cref{sec:completion_script}.

Below the common options an overview of the currently configured sequence of test patterns
is displayed. Only the durations are displayed, the configured faults
are omitted in this view.

Test patterns can be configured by clicking the \textit{Edit} button
next to the \textit{Duration~T2} column. This opens a dialog box where
the timer values can be changed, the individual \ac{FIU}s configured, and
reset and trigger options enabled.

Test patterns can be appended at the end by \textit{Append Test} and
deleted individually by clicking the respective \textit{Delete Test} button.
Appended tests are always ``benign'', i.e., their durations are set to a workable minimum depending on the design frequency, the \ac{FIU}s are configured to be fault-free (``NONE''), and the trigger and reset options are disabled.

\subsubsection{Manual}

\Cref{fig:manual_download_gui} shows the \textit{Main Area} with
\textit{Manual} mode selected. Here, the parameters of a single fault
configuration can be entered. By clicking the \textit{Start} button with
manual mode selected, the currently entered configuration is sent to
the fault injection logic where it is immediately activated when the
logic is ready.

The optional \ac{FIU} names are displayed next to the respective index to 
help identifying the \acp{FIU}. The net the respective \ac{FIU} is attached
to is displayed as a tooltip for the \ac{FIU}'s label and the drop-down
menus to select faults.

As can also be seen in the Figure, the
hardware configuration information from the \textit{\ac{FIJI} Settings} file
is incorporated into the user interface. In the hardware configuration loaded
for the screenshot, the \acs{FIJI}-to-\acs{DUT} reset feature was turned off, and
the \textit{\ac{FIJIEE} GUI} notifies the user by graying-out the respective
checkbox and showing an explanation next to it.

\begin{figure}[!htb]
\centering
\includegraphics[width=0.85\linewidth]{img/fiji_ee_manual_screenshot.png}
\caption{\ac{FIJIEE} GUI: Manual Mode}
\label{fig:manual_download_gui}
\end{figure}

\subsubsection{Random}

For the random test the user can determine the global probability of the various fault models.
These probabilities are then used to determine the state of the \acsp{FIU} in each message sent.

There are two modes of operations that can be selected by the (``multiple faults per pattern'') option:
\begin{description}
    \item[Single Faulty \acs{FIU}]
        If the option is disabled then for each configuration message a single \ac{FIU} is selected among all \acp{FIU} to be (potentially) faulty in one (randomly selected) configuration pattern.
        For the selected pattern this \ac{FIU} is then armed according to the global probabilities.
        All other \acp{FIU} and the selected one in all but the selected pattern are set to \textit{No Fault}.
    \item[Multiple Faults]
        If the option is enabled then all \acp{FIU} in all patterns are subject to (potential) faults.
        A chance for multiple faulty signals at any given point are the consequence.
\end{description}

The GUI elements for configuring the \textit{Random} test mode can be
seen in \Cref{fig:random_download_gui}.
In addition to those parameters already described in \Cref{sec:sequence}, the following parameters are configurable:

\begin{itemize}
    \item The probability of the different fault models in each randomization operation.
          The difference of the sum of all fault probabilities to 1 is the probability
          that a \acs{FIU} is fault-free (i.e., set to \textit{NONE}).
    \item If it is possible that more than one FIU is configured to be faulty in any
          pattern as explained above.
    \item If the FIJI-to-\acs{DUT} reset signal shall be asserted in the initial
          fault configuration (e.g., to bring the \ac{DUT} into a defined state prior
          to fault injection)
\end{itemize}

\begin{figure}[!htb]
\centering
\includegraphics[width=0.85\linewidth]{img/fiji_ee_random_screenshot.png}
\caption{\ac{FIJIEE} GUI: Random Mode}
\label{fig:random_download_gui}
\end{figure}

\subsubsection{Completion Script}
\label{sec:completion_script}
In \textit{Sequence} and \textit{Random} mode a Perl script can be selected to be executed upon test completion.
An example script is included that sends an email containing the test results, all log messages and the test triggering the fault if any.

The \textit{Completion Script} is given the following parameters (in this order):

\begin{itemize}
    \item The reply of the respective \texttt{FIJI::Downloader}'s \texttt{download\_*} function, i.e., an error message if something unexpected happened, or a hash containing the following fields:
    \begin{itemize}
        \item \texttt{msg\_type}: the type of the last received message.
            One of \texttt{CONF\_DONE}, \texttt{READY} \texttt{UNDERRUN}.
        \item \texttt{fault\_detect}: a hash indicating if a fault detection signal has been asserted.
            Contained keys: \texttt{ANY}, \texttt{1}, \texttt{2}
        \item \texttt{error}: another hash indicating any error flags sent by the \ac{FIC}.
            Contained keys: \texttt{ANY}, \texttt{I} (ID mismatch), \texttt{C} (CRC error), \texttt{U} (malformed UART frame)
    \end{itemize}
    \item The start and end time of the test execution in fractional seconds since the Unix epoch.
    \item A reference to the faulty \texttt{FIJI::Tests}.
    \item All available log messages as display in the respective UI.
\end{itemize}

Any return value of the script is ignored.

\subsubsection{Export}

Once a fault injection test has been completed (either a complete sequence 
has been downloaded, a fault detect line reported an error, the user aborted
the sequence or manual downloading, or a transmission error occurred), this
test run can be exported for reproducing it at a later time in hardware or
in simulation.

This is done using the \textit{Export} menu entry in the \textit{Control Area}
(marked as (1) in \Cref{fig:download_gui}). This menu provides the
following entries:

\begin{itemize}
    \item \textit{As Test Configuration}: This will open a dialog to save
          the completed test run as a \textit{\ac{FIJI} Tests} file for 
          later re-execution.
    \item \textit{As RTL Simulation Template}: This will export the current
          test run to VHDL-2008 and System Verilog templates for \ac{RTL} simulation.
          This HDL description contains (1) fault injection logic templates
          which have to be integrated into the various \ac{RTL} modules manually,
          and (2) a scheduling process which controls these templates using
          hierarchical identifiers whose path also has to be adapted.
    \item \textit{As VHDL architecture for gate-level simulation}: This
          generates a simulation-only architecture for the top-level
          fault injection module. It is intended to be used as a replacement
          for the \texttt{struc} architecture when synthesizing the
          design for fault injection tests.
          Thus, for performing a gate-level simulation with the recorded
          faults, a testbench for the fault injection wrapper
          module needs to be generated, and compiled with the modified netlist,
          the top-level entity of the fault injection logic as well as
          its dependencies (packages) and the exported simulation architecture.
\end{itemize}

\subsection{Gate-Level Simulation}

As mentioned above, \ac{FIJI} helps with re-executing a test that ran in actual
hardware as a gate-level simulation. To that end, \textit{\ac{FIJIEE}} tools
are able to export the executed tests as a VHDL architecture for the top-level
entity of the fault injection logic. This architecture replaces the \ac{FIC} and
the \acp{FIU} with a simulation-only description which sets the modified
net outputs according to the timing of the test run previously executed in
hardware.

To perform a gate-level simulation with a previously executed test sequence,
the following steps need to be performed:

\begin {enumerate}
    \item Perform a ``Sequence'' or ``Random'' test using a \textit{\ac{FIJIEE}} tool
       until a fault is detected.

    \item  Export the test run to a ``VHDL architecture for gate-level simulation''.

    \item  Set up a work library and compile the FPGA primitives required
           for gate-level simulation into that library using a logic simulator.

    \item Compile the following source files into the work library:
        
        \begin{itemize}
            \item \texttt{<path/to/fiji>/hw/rtl/fault\_selection\_type\_pkg.vhd}:

                  the 'fault selection type' required by the fault injection logic's toplevel

            \item the VHDL configuration package generated by \textit{\ac{FIJI} instrument}
        
            \item the following required parts of the FIJI HDL source:

                \begin{itemize}
                    \item \texttt{<path/to/fiji>/hw/rtl/private\_config\_pkg.vhd}
                    \item \texttt{<path/to/fiji>/hw/rtl/fault\_injection\_top\_pkg.vhd}
                    \item \texttt{<path/to/fiji>/hw/rtl/fault\_injection\_top\_.vhd}
                \end{itemize}

            \item the simulation architecture exported by \textit{\ac{FIJIEE}}

            \item the instrumented netlist generated by \textit{\ac{FIJI} instrument}

            \item the VHDL wrapper generated by \textit{\ac{FIJI} instrument}

            \item a testbench which instantiates the VHDL wrapper and supplies stimuli to its inputs
        \end{itemize}

    \item Run the simulation until the simulation architecture of the
          fault injection logic reports the end of the test run.
\end{enumerate}