13.3.4. Basic examples

Simplest

run_afni_tests.py local

Under the hood this is running something like: “cd /path/to/tests/directory pytest scripts –tb=no –no-summary –show-capture=no” (where scripts is a directory containing python test scripts, and pytest is a python tool used for running a test suite)

The above test run (by this I mean ‘execution of tests’) implicitly uses the AFNI software found first on the PATH.

Install dir: Explicitly defining the installation directory

run_afni_tests.py --abin=~/abin local

The above will make sure that the intended version of AFNI programs are found (including the python libraries contained in the afnipy subdirectory). It will raise an error if the afnipy package is installed into your current interpreter.

CMake: Making use of the cmake build system

In order to facilitate the iterative process of changing code and running tests you can run the tests directly on the executables/scripts in the build/source trees. When you run the following it will initially take a long time because it needs to build the full suite of AFNI tools. Subsquently, minimal rebuilds occur: i.e. any changes in the source tree will trigger a rebuild only of binaries that are affected by the change (or at least that is the hope).

mkdir /tmp/build
run_afni_tests.py --build-dir=/tmp/build local

The above defines an empty directory as the build directory. The cmake tool will write its output to this directory. This output consists of the output of the three stages of cmake execution: configure time, generate time, build time.

In brief, configure time is when cmake detects the details of your system in order to adapt its output appropriately, generate time is when cmake actually generates a build system (this can be a make, ninja, or some other build system), and finally build time makes use of the aforementioned build system to generate the projects executables.

13.3.5. pytest: Important! It’s pytest under the hood

While this wrapper script interacts with the build systems and facilitates testing with containers it is ultimately just a wrapper around the pytest tool that is used to execute AFNI’s tests. Keeping this in mind will greatly aid you in customizing test runs. Displaying the help for the pytest tool (including the options specifically defined for AFNI in the “custom-options” section) can be done by running:

run_afni_tests.py --extra-args='--help' local

In the above call for help you will hopefully notice that passing arguments to pytest is done by using a quoted string to the –extra-args option. All options that you might think to pass are displayed in pytest’s help output. Just as there is a custom options section for AFNI, other plugins modify the available options for the pytest tool. The next example shows this.

Resources: Tuning resource usage

The pytest-parallel package is a plugin that works with pytest to enable parallel execution of the test suite. If installed then the following pattern can be used to tune the resources used for testing:

run_afni_tests.py --extra-args='-n=4' local

Assuming you just want to use every core the following convenience of run_afni_tests.py allows you do this.

run_afni_tests.py --use-all-cores local

Test selection: Subsetting the tests that are executed

There are different ways of tuning the tests that are executed. The most basic is to specify the exact test file that should be run. The file path should be specified relative to the tests directory in the AFNI repo:

run_afni_tests.py --file scripts/test_ptaylor.py local

Another way of subsetting the tests executed is to use the -k flag of the pytest tool:

run_afni_tests.py --extra-args='-k mask' local

For convenience this functionality can be accessed directly by the -k flag of run_afni_tests.py

run_afni_tests.py -k mask local

Another nice trick worth knowing is that you can re-execute all failed tests from the last test run (like -k, it is an option of pytest and run_afni_tests.py):

run_afni_tests.py --lf local

Manual pytest: Manually specifying pytest arguments

You can drop all behavior that is enforced by run_afni_tests.py and specify your own pytest options:

run_afni_tests.py --overwrite-args='--verbose' local

13.3.6. Reporting: Modifying the reporting of a tests run

Verbosity: Altering test reporting verbosity

Verbosity at the terminal can be altered as described below but remember that for any given test the stdout, stderr, as well as the context for the executed are all recorded in the captured_output subdirectory for each test.

You can alter the verbosity of the test output:

run_afni_tests.py -v verbose local

This can be increased a few levels of verbosity:

run_afni_tests.py -v diarrhetic local

13.3.7. Debugging: When you have failing tests

There are some python debugging tools that can be useful for exploring test failures. For example the –debug option will drop you into a pdb console (python debugger console) at the test failure giving you access to the full call stack. This can be very useful for figuring what went wrong and where:

# Note that --debug does not work in combination with the container subcommand
run_afni_tests.py --debug local

The above command makes use of the –pdb pytest option under the hood. Additionally –trace will allow you to step line by line through tests. It might be useful for your purposes…

Shell: Debugging using the shell

For many of the tests in the suite, the actual test boils down to the execution of a simple shell command (with some subsequent comparisons with known expected output). In these situations it can be useful to take a step out off pytest/run_afni_tests.py and reproduce the problem using a simple shell command. There are various bits of trickiness that run_afni_tests.py and the pytest configuration attempts to avoid for the user so this will not always work but the basic approach to doing this is:

  1. Run the command in verbose mode. It is a good idea to try to be specific here. Use the approaches in a previous section on subsetting the tests that are executed to reduce the amount of relevant output. For our purposes let’s pretend we have a single test that is failing and we can rerun only that using the –lf selector for last failed tests.

    run_afni_tests.py –lf -vvvvv local

  2. You may have to modify the PATH to reproduce the test. For example, when using the –build-dir option, you will have to modify the PATH variable according to what is printed to stdout during typical execution with run_afni_tests.py.

  3. Search for the command executed. This is logged prior to execution. It will start with a command to cd into the tests directory.

  4. Consider whether output already exists. You may want to create a new output directory to avoid issues here. Or just remove files that have been generated already by the test execution that failed.

13.3.8. Using docker for environment

This provides immmense advantages including:

  • encapsulation of almost all dependencies (besides docker itself, python 3, and the python docker package)

  • reliable comparisons between linux system and MacOS

  • having independence between the test run and previous activity

It’s usage can be a little confusing. There is a lot going on… reading this section to familiarize yourself with the higher level details will be extremely helpful.

Local dirs: Host source and build directory

This is ideal. The source and build directory on your local host are used within the container. It means that small changes to the source will trigger minimal rebuilds within the container because the build directory timestamps will be closer to realtime (as opposed to when the image was originally built). Caveats here are that you need to initially mount an empty build directory (not one from a build in the local environment) and that on OSX using this configuration will take a substantial performance hit due to IO on mounted volumes. You should only suffer the first time though. After that, minimal rebuilds will make this issue less significant.

run_afni_tests.py --build-dir /path/to/existing/directory container --source-mode=host

Pre-existing: Reuse a build in the container.

This will reuse the build directory in the container to build everything from the source on the local host. It will be slow because likely a substantial rebuilding of the binaries will occur. For a once off test, it allows you to test your most recent source tree inside a container though, and you may still get some speed up from using the pre-existing build (compared to the previous configuration described).

run_afni_tests.py container --reuse-build

Fully contained: Build/test the image locally

This is a nice sanity check that your current code passes tests independent of your current environment. To do this you would run the following from the root directory in the afni git repository:

docker build -f .docker/afni_dev_base.dockerfile -t afni/afni_dev_base .
docker build -f .docker/cmake_build.dockerfile -t my_image .
run_afni_test.py container --image-name my_image

Mount tests: Running current test suite in pre-existing container

This avoids permissions/performance issues but still allows for some flexibility. It is useful for iterating on test code while testing the binaries that were installed into the container.

run_afni_tests.py container --source-mode=test-code

13.3.9. Getting help with the setup for testing

The readme in the tests directory for setting up your environment for testing can be viewed with the following command:

./run_afni_tests.py --installation-help