Testing

Continuous Integration (CI) Testing

In every pull request (PR), the ci.yml GitHub workflow runs the following tests:

• Linting - To ensure new scripts and packages meet our code and structure requirements we run two linters:
  • The PowerShell script scripts\test\lint.ps1, which runs PSScriptAnalyzer, a static code checker for PowerShell.
  • The Python script scripts\test\lint.py which checks that some of our coding conventions are followed.
• Package installation - To ensure packages install successfully we run the PowerShell script scripts/test/test_install.ps1 which uses choco install to install all modified packages.

The tests need to succeed to merge the PR. Once the PR is merged, the ci.yml GitHub workflow automatically pushes the packages to the MyGet vm-packages packages feed. This means they're immediately available to be installed in everyone's VM.

Local testing

When adding or modifying a package (and when reviewing a package addition/modification), it is important to test the changes locally. Our test suite runs several linters and tests that the package is installed, but it doesn't start the tool to ensure it works correctly. Uninstalling is best-effort and we are not testing uninstalling in our our CI, please test it also locally.

The best way to test packages is to install the installer.vm package with few or no packages in a fresh Windows VM. An easy way to achieve this is to perform a minimal installation of FLARE-VM.

Minimal FLARE-VM installation

Follow the following steps to perform a minimal FLARE-VM installation (with very few packages) that you can use for testing.

1. Prepare a Windows 10 VM and disable Windows Updates, Tamper Protection and Windows Defender as explained in the Pre-installation section of the FLARE-VM README. Take a snapshot named BUILD-READY.

2. Download the FLARE-VM configuration file and modify it to remove all <package /> entries except <package name="hashmyfiles.vm"/> and <package name="pma-labs.vm"/>. hashmyfiles.vm installs HashMyFiles, a small utility useful to calculate SHA256 hashes (needed to create some packages). pma-labs.vm downloads binaries that can be useful for testing. These two packages do not install any other tools or dependencies that could affect testing.

   <packages>
       <package name="hashmyfiles.vm"/>
       <package name="pma-labs.vm"/>
   </packages>
   

3. Execute the FLARE-VM PowerShell installation script in an admin PowerShell console using the modified config file with only the packages hashmyfiles.vm and pma-labs.vm as explained in the FLARE-VM installation section of the FLARE-VM README: .\install.ps1 -customConfig <config.xml> -password <password> -noWait -noGui

   

4. After the FLARE-VM installation finishes (which may take some time), copy the test_install.ps1 script to the Desktop and create a packages folder in the Desktop. Open an admin PowerShell console, set the current working location to the Desktop and write (but do not execute): ./test_install.ps1.

   

5. Take a testing snapshot.

Every time you want to test a package or packages, restore the testing snapshot, copy the package/s to the packages folder in the Desktop and run the test_install.ps1 script (by just pressing Enter in the PowerShell console). After the installation completes, ensure the installed tools work as expected. When applicable, test also that the package uninstalls correctly. The test_uninstall.ps1 script may be useful to test uninstallation.

If you need to test the same package/s again, restore to the testing snapshot even if the first installation failed to ensure no artifacts remain that could affect the testing.

Notes

Using a shared folder to copy the packages over

You can store the packages to test in a shared folder and create a link to it in the Desktop to speed the testing process up.

1. Create a testing shared folder. In VirtualBox, you need to create the folder on the host and add it to the Shared Folders in the VM settings. The shared folder will appear in the VBOXSVR virtual network server.

   

2. Create a packages folder inside the testing shared folder.

   

3. Copy the packages folder and paste it as a shortcut in the Desktop. Rename it to packages.

   

4. [Optional] You may want to create a shortcut in the Desktop to the testing shared folder for faster access as well.

5. Take a testing snapshot.

Now you can copy directly packages to test to the packages shared folder, restore to the testing snapshot, and test the packages running the test_install.ps1 script (by just pressing Enter in the PowerShell console).

Testing hashmyfiles.vm or pma-labs.vm

If you need to test hashmyfiles.vm or pma-labs.vm, you would need to perform a minimal installation with different simple packages or without any packages.

Testing tools with dependencies

When testing packages that require extra tools or depend on packages that take long to install, it is recommended that you install all the requirements first and take a snapshot before installing the package that you want to test. This way you can revert to this snapshot if you need to adapt and test the package several times without having to install the requirements again.

To install a package or packages, run choco install <package_name_or_names>.

Examples:

Testing IDA related packages

Before testing IDA related packages, such as IDA plugins:

1. Install libraries.python3.vm to install Python 3 and several useful libraries and idafree.vm to install IDA free, which includes some improvements that works with IDA Pro as well (such as the Open with IDA right click shortcut):

   choco install libraries.python3.vm idafree.vm
   

2. Install IDA Pro (using your installer) which will find the installed Python

Testing Node tools installed with npm

Install nodejs.vm: choco install nodejs.vm.