Skip to content

Commit

Permalink
Unit test update
Browse files Browse the repository at this point in the history
  • Loading branch information
@adrian-soch
adrian-soch committed Apr 2, 2024
1 parent 41c5116 commit 277a7b3
Show file tree
Hide file tree
Showing 2 changed files with 94 additions and 125 deletions.
Binary file modified docs/VnVPlan/VnVPlan.pdf
View file
Binary file not shown.
219 changes: 94 additions & 125 deletions docs/VnVPlan/VnVPlan.tex
View file
Original file line number Diff line number Diff line change
Expand Up @@ -307,11 +307,9 @@ \subsection{Automated Testing and Verification Tools}
\subsection{Software Validation Plan}

For the scope of this project, software validation will consist of benchmarking against existing
algorithms. The Root Mean Squared Error (RMSE) of \progname{} should lie within $\epsilon$ of the
RMSE of attitude estimators implemented by \citet{broad_code}. The data will come from the BROAD
dataset \citep{broad}.

The derivation of RMSE can be found in the Section \ref{sec:calc}.
algorithms. The \% difference of the Root Mean Squared Error (RMSE) of \progname{} and
\href{https://ahrs.readthedocs.io/en/latest/filters/madgwick.html}{AHRS} should be $\leq \epsilon$.
The data will come from the BROAD dataset \citep{broad}. The derivation of RMSE can be found in the Section \ref{sec:calc}.

\wss{If there is any external data that can be used for validation, you should point to it here. If
there are no plans for validation, you should state that here.} \wss{You might want to use review
Expand Down Expand Up @@ -450,9 +448,11 @@ \subsubsection{Calculations} \label{sec:calc} This section covers requirements R
\end{tabular} & - \\ \hline
2 & \begin{tabular}[c]{@{}l@{}}Call ``update" and loop through \\ measurements in Table
\ref{tab:gt_data} \end{tabular} & Assert quaternion magnitude is 1. \\ \hline
3 & \begin{tabular}[c]{@{}l@{}}Calculate RMSE for $e_q, e_h, e_i$\\ vs. ground truth in
table \ref{tab:gt_label}\end{tabular} & \begin{tabular}[c]{@{}l@{}}Assert
average error is within\\ the tolerance, $\epsilon$.\end{tabular} \\ \hline
3 & \begin{tabular}[c]{@{}l@{}}Calculate RMSE of $e_i$ for
\href{https://ahrs.readthedocs.io/en/latest/filters/madgwick.html}{AHRS}\\and \progname{}
using table \ref{tab:gt_label}\end{tabular} &
\begin{tabular}[c]{@{}l@{}}Assert that the \% difference between \\ methods $\leq
\epsilon$.\end{tabular} \\ \hline
\end{tabular}
\end{table}

Expand All @@ -479,9 +479,11 @@ \subsubsection{Calculations} \label{sec:calc} This section covers requirements R
2 & \begin{tabular}[c]{@{}l@{}}Call ``update" and loop through \\ \{accel, gyro\}
measurements in Table \ref{tab:gt_data} \end{tabular} & Assert output quat is normalized \\
\hline
3 & \begin{tabular}[c]{@{}l@{}}Calculate RMSE of $e_q, e_h, e_i$\\ vs. ground truth in table
\ref{tab:gt_label}\end{tabular} & \begin{tabular}[c]{@{}l@{}}Assert average
error is within\\ the tolerance $\epsilon$.\end{tabular} \\ \hline
3 & \begin{tabular}[c]{@{}l@{}}Calculate RMSE of $e_i$ for
\href{https://ahrs.readthedocs.io/en/latest/filters/madgwick.html}{AHRS}\\and
\progname{} using table \ref{tab:gt_label}\end{tabular} &
\begin{tabular}[c]{@{}l@{}}Assert that the \% difference between \\ methods $\leq
\epsilon$.\end{tabular} \\ \hline
\end{tabular}
\end{table}

Expand Down Expand Up @@ -515,9 +517,10 @@ \subsubsection{Accuracy}

\item[\refstepcounter{testnum} \Tthetestnum \label{t:a1}:] \textbf{test-a1} \\

Type: Automatic
Type: Manual

This test refers to Section \ref{sec:calc}.
This test refers to Section \ref{sec:calc}. The accuracy test will be conducted when changes to
the logic or calculation components of the program are made. See \url{https://github.com/adrian-soch/attitude_check/blob/main/scripts/compare_results.py}.
\end{enumerate}

\subsubsection{Understandability}
Expand Down Expand Up @@ -557,6 +560,8 @@ \subsubsection{Maintainability}
How test will be performed: If a likely change needs to be added, measure the development time
and compare it to the total development time. If it is less than the threshold specified in the
SRS, this test passes.

This test will not be explicitly conducted.
\end{enumerate}

\subsubsection{Portability}
Expand All @@ -573,9 +578,8 @@ \subsubsection{Portability}

Output/Result: Result of compilation and execution.

How test will be performed: Build the project, initialize and input 1 set of measurements. Test
is successful if the build has no errors and if the code produces the same result on Windows 11,
macOS, and Ubuntu.
How test will be performed: Build the project on Ubuntu and MacOS. Build and execute \progname{}
on a microcontroller with an IMU/MARg sensor.
\end{enumerate}

\subsection{Traceability Between Test Cases and Requirements}
Expand All @@ -601,140 +605,105 @@ \subsection{Traceability Between Test Cases and Requirements}

\section{Unit Test} \label{sec:unit_test}

\wss{This section should not be filled in until after the MIS (detailed design document) has been
completed.}

\wss{Reference your MIS (detailed design document) and explain your overall philosophy for test case
selection.}

\wss{To save space and time, it may be an option to provide less detail in this section. For the
unit tests you can potentially layout your testing strategy here. That is, you can explain how
tests will be selected for each module. For instance, your test building approach could be test
cases for each access program, including one test for normal behaviour and as many tests as needed
for edge cases. Rather than create the details of the input and output here, you could point to the
unit testing code. For this to work, you code needs to be well-documented, with meaningful names
for all of the tests.}
This section provides details on the white box unit testing for \progname{}.

\subsection{Unit Testing Scope}

This will be updated when the MG and MIS are complete.

\wss{What modules are outside of the scope. If there are modules that are developed by someone
else, then you would say here if you aren't planning on verifying them. There may also be modules
that are part of your software, but have a lower priority for verification than others. If this
is the case, explain your rationale for the ranking of module importance.}
The scope of the unit tests will not include testing the matrix math functionality of the Eigen
library.

\subsection{Tests for Functional Requirements}

This will be updated when the MG and MIS are complete.

\wss{Most of the verification will be through automated unit testing. If appropriate specific
modules can be verified by a non-testing based technique. That can also be documented in this
section.}
The tests for each module will be detailed here, each test will be part of an automated unit test
suite.

\subsubsection{Module 1}
\subsubsection{Quaternion Module}

\wss{Include a blurb here to explain why the subsections below cover the module. References to the
MIS would be good. You will want tests from a black box perspective and from a white box
perspective. Explain to the reader how the tests were selected.}
This module contributes to R1, R2, R3, and R4. Tests can be found in
\href{https://github.com/adrian-soch/attitude_check/blob/main/test/quaternion_test.cpp}{quaternion\_test.cpp}
The test names are related to the function that it tests. The tests are self documenting, since the
assertions and expected values are explicitly coded. Some tests are repeated for single and double
precision floating point values.

\begin{enumerate}

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Inputs: Invalid time} \\

\textbf{Initial State:} ...

\textbf{Input:} ...

\textbf{Output:} ...

\textbf{Test Case Derivation:} ...

\textbf{How test will be performed:} ...

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Inputs: Invalid mag ref} \\

\textbf{Initial State:} ...

\textbf{Input:} ...

\textbf{Output:} ...

\textbf{Test Case Derivation:} ...

\textbf{How test will be performed:} ...

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Inputs: Invalid gamma} \\

\textbf{Initial State:} ...

\textbf{Input:} ...

\textbf{Output:} ...

\textbf{Test Case Derivation:} ...

\textbf{How test will be performed:} ...

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Inputs: Invalid output type} \\

\textbf{Initial State:} ...

\textbf{Input:} ...

\textbf{Output:} ...

\textbf{Test Case Derivation:} ...

\textbf{How test will be performed:} ...

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Inputs: Valid} \\

\textbf{Initial State:} ...

\textbf{Input:} ...

\textbf{Output:} ...

\textbf{Test Case Derivation:} ...

\textbf{How test will be performed:} ...

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Invalid initialization}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Conjugate}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Product}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Norm}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Scalar product}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Subtract}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Add}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Subtract Equals}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Equals}
\end{enumerate}

\subsection{Tests for Nonfunctional Requirements}
\subsubsection{Utilities Module}

\wss{If there is a module that needs to be independently assessed for performance, those test cases
can go here. In some projects, planning for nonfunctional tests of units will not be that
relevant.}

\wss{These tests may involve collecting performance data from previously mentioned functional
tests.}

This will be updated when the MG and MIS are complete.

\subsubsection{Module TBD}
This module contributes to R2. Tests can be found in
\href{https://github.com/adrian-soch/attitude_check/blob/main/test/utilities_test.cpp}{utilities\_test.cpp}
The test names are related to the function that it tests. The tests are self documenting, since the
assertions and expected values are explicitly coded.

\begin{enumerate}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Euler to quat (double)}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Euler to quat (float)}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Rotation Matrix to quat (double)}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Rotation Matrix to quat (single)}
\end{enumerate}

\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{test-id1} \\

Type: \wss{Functional, Dynamic, Manual, Automatic, Static etc. Most will be automatic}
\subsubsection{Initializers Module}

Initial State:
This module contributes to R2. Tests can be found in
\href{https://github.com/adrian-soch/attitude_check/blob/main/test/initializers_test.cpp}{initializers\_test.cpp}
The test names are related to the function that it tests. The tests are self documenting, since the
assertions and expected values are explicitly coded.

Input/Condition:
\begin{enumerate}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Accel to quat}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Mag to quat}
\end{enumerate}

Output/Result:
\subsubsection{Attitude Check Module}

How test will be performed:
This module contributes to R1, R2, R3, R4, and R5. Tests can be found in
\href{https://github.com/adrian-soch/attitude_check/blob/main/test/attitude_check_test.cpp}{attitude\_check\_test.cpp}
The test names are related to the function that it tests. The tests are self documenting, since the
assertions and expected values are explicitly coded.

\begin{enumerate}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Invalid Initialization}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Invalid call to update}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Call with zero mag vector}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Call with zero gyro vector}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Call with zero acc vector}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Set quaternion}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Set gain}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Get gain}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Get Initial orientation}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Check MARG calculation}
\item[\refstepcounter{utestnum} \UTthetestnum:] \textbf{Check IMU calculation}
\end{enumerate}

\subsection{Tests for Nonfunctional Requirements}

There are no unit tests for nonfunctional requirement.

\subsection{Traceability Between Test Cases and Modules}

This will be updated when the MG and MIS are complete.
\begin{table}[h]
\centering
\caption{Traceability Between Test Cases and Modules}
\vspace{3mm}
\label{tab:my-table}
\begin{tabular}{l|c}
Module & Tests \\ \hline
Quaternion & UT1 - UT9 \\
Matrix Math & None \\
Utilities & UT10 - UT13 \\
Initializers & UT14 - UT15 \\
Attitude Check & UT16 - UT 26
\end{tabular}
\end{table}

\wss{Provide evidence that all of the modules have been considered.}

Expand All @@ -759,7 +728,7 @@ \subsection{Symbolic Parameters}
\begin{tabular}{|l|l|}
\hline
\textbf{Param} & \textbf{Value} \\ \hline
$\epsilon$ & 5 \% \\ \hline
$\epsilon$ & 3 \% \\ \hline
\end{tabular}
\end{table}

Expand Down

0 comments on commit 277a7b3

Please sign in to comment.