Regression Testing

carputils provides the carptests executable in the bin subfolder, while enables the automatic running of sets of tests and the generation of reports comparing the results against reference solutions.

Consider adding the bin subfolder of carputils to your $PATH environment variable to make carptests runnable from anywhere on the command line.

Structure of Repositories

The testing framework is composed of several git repositories:

  • carputils - A python package providing common functionality for all examples

  • The example repositories, which are each a set of python scripts which set up and run CARP simulations, including

  • devtests - smaller CARPentry simulation examples

  • benchmarks - larger CARPentry simulation examples

  • The reference solution repositories, which contain previously computed solutions to the above examples that are known to be correct, including

    • devtests-reference
    • benchmarks-reference

Why init.py?

The init.py files in devtests and benchmarks may seem odd, however they are required by the testing framework. carptests uses the python package hierarchy to find and import individual tests, so for that reason the example repos must be python packages. Python determines that a directory is a package if it contains a file named init.py. This can be used to control how the package is imported, however for our purposes we only need to add an empty init.py file in all directories from the top level of the package (devtests/init.py, for example) down to the level of the directory where there is a test we want to run.

Acquiring Existing References

To run regression tests, you will need to acquire the reference solutions. Make a directory to store them and clone the reference solution repositories there:

    mkdir /data/carp-regression-reference
    cd /data/carp-regression-reference
    git clone git@bitbucket.org:carpusers/devtests-reference.git devtests
    git clone git@bitbucket.org:carpusers/benchmarks-reference.git benchmarks

Make sure that the cloned respository name matches that of the corresponding test repositories above. Then, set the parent directory as the REGRESSION_REF setting in settings.yaml:

    REGRESSION_REF: /data/carp-regression-reference

Optionally, set the REGRESSION_TEMP and REGRESSION_PKG settings in settings.yaml, as described in the file template.

Reference Repository Layout

carptests automatically determines where to find the reference solutions based on the REGRESSION_REF setting in the carputils settings.yaml file, the name of the module that the test was found in, and the name of the test. To ensure that the reference solution files are exactly where they are expected to be, it is strongly encouraged that users generate the reference solution with the --refgen parameter to carptests, as described in carptests-refgen. This not only ensures that the reference is stored in the right place, but also that it is generated with exactly the inputs parameters expected by the test.

Running Regression Tests

Running the executable without arguments will by default run all tests in the module(s) specified in REGRESSION_REF in settings.yaml:

    carptests

A summary of the daily results can be found here.

To override this behaviour, specify one or more modules on the command line. For example, to run all mechanics examples in devtests and all tests in benchmarks, run:

    carptests devtests.mechanics benchmarks

Tests can also have case-insensitive tags assigned. Tags are defined in the carputils.testing.tag module, and summarised in the automatic documentation. To run, for example, only fast tests taking less than 10 seconds:

    carptests --tags fast

On completion of all tests, a report is generated summarising the results. For more details on the command line interface, run:

    carptests --help

Generating Reference Solutions

To generate and store any files needed for testing in the relevant repository, run carptests with the --refgen optional argument. You will probably want to specify a single example to generate the reference solutions for to avoid overwriting the entire suite's reference solutions. For example, to generate the reference solutions for the mechanics ring example in devtests (devtests/mechanics/ring), run:

    carptests devtests.mechanics.ring --refgen

This will run the test(s) specified in the mechanics ring example and copy the files needed for comparison in the test to the correct location in the reference solution repository. A summary is generated of the run tests, including a list of the generated reference solution files:

    ===================================================================
                                  SUMMARY
    ===================================================================

      Tests run .................. 1
      Reference files generated .. 1
      Runtime .................... 0:00:17.049971

      Files generated:
        /data/carp-test-reference/devtests/mechanics.ring.run/neumann/x.dynpt.gz

    ====================================================================

If you are satisfied that this generated solution is the 'correct' one, that others should compare against when running this test, you need to commit and push this new or modified file. In the above example:

    cd /data/carp-test-reference/devtests/mechanics.ring.run/neumann/
    git add x.dynpt.gz
    git commit -m "Updating the mechanics ring test ref. solution"
    git push # To push your commit to bitbucket and share with others

Defining New Tests

The 'simple bidomain' example in the devtests repository (devtests/bidomain/simple) defines some simple tests and can be used as a starting point for new examples with tests.

The testing framework views the tests' directory structure as a Python package and searches the package hierarchy to find and run tests which have been defined.

Tests are defined by assigning the variable __tests__ in the top level
namespace of a run script, which must be a list of

class:carputils.testing.Test objects. This can be placed anywhere in the source file outside a function or __name__ == '__main__' block, but it is suggested to put it immediately before the __name__ == '__main__' block, and after the definition of the run function described in :ref:examples::

from carputils import tools from carputils.testing import Test

@tools.carpexample() def run(args, job): pass # Placeholder

test1 = Test(...) test2 = Test(...) tests = [test1, test2]

if name == 'main': run()

The :class:carputils.testing.Test object takes at least two arguments:

  • The name of the test
  • The function to run to execute the simulation

Additionally, tests will usually take at least a third optional argument:

  • A list of command line options to run the test with.

So, if you were to normally run a test with the command line:

.. code-block:: bash

./run.py --experiment neumann --np 8

You might configure the test::

test1 = Test('neumann_np8', run, ['--experiment', 'neumann', '--np', 8])
The function to run (here just called run) should be as explained in

ref:examples::

@tools.carpexample() def run(args, job):

Run the simulation

pass # Placeholder

Adding Checks

You must also tell the framework how to validate the simulation output against a reference solution. Generally, you will do this by comparing an output file against a reference, and computing some kind of error norm.

The :mod:carputils.testing.checks module defines some functions for comparing
igb files, :func:carputils.testing.max_error and

func:carputils.testing.error_norm. These functions calculate a scalar error value, which the testing framework compares against the test developer's preassigned tolerance. A check is added to a test by::

from carputils.testing import max_error test1.add_filecmp_test('x.dynpt.gz', # Output file to compare max_error, # Error calc method 0.001) # Error tolerance

These simple error methods should cover most use cases, though you may easily create your own by passing a function that takes the reference filename as the first argument and the temporary test output at runtime as the second argument, and returns an error value.

The error tolerance value is only used to determine test success. For more complex control of test success, you may alternatively pass as the tolerance argument a function which takes the return value of the error function and returns a boolean.

Important Considerations

  • The directory your test is in must be part of the Python package. To ensure this is the case, there must be an empty file called __init__.py in the same directory as your test's run script, and any intermediate directories from the top level package down.

Reference Generation

Once your tests are defined, you will need to share any reference simulation
results required for comparison in the tests. To do so, run the carptests
executable with the --refgen parameter (probably with the module of your
example specified to avoid running all examples) as described in
ref:carptests-refgen.

Optional Arguments

Tags

Tests can optionally set one or more tags, for the purposes of categorisation of simulations. Standard tags are as described in carputils.testing.tag, but test developers can also add their own limited-use tags with the add method::

from carputils.testing import tag
tag.add('my_special_tag')

Tags are assigned to test by supplying a list of tags as the tags keyword argument::

test1 = Test('mytest', run, ['--argument'],
             tags=[tag.SERIAL, tag.FAST,
                   tag.MY_SPECIAL_TAG]) # tags are case insensitive

Tests can then be filtered at runtime with the --tags command line argument, as described in :ref:carptests-usage.

Reference Directory

If you want to validate multiple tests against the same reference solution, consider use of the refdir keyword argument.

By default, reference solutions are stored in the directory:

.. code-block:: bash

REGRESSION_REF/package/module/testname

where REGRESSION_REF is replaced by the value set in the carputils settings.yaml file. For example, for the test neumann in devtests/mechanics/ring/run.py, this will be:

.. code-block:: bash

REGRESSION_REF/devtests/mechanics.ring.run/neumann

To force a test of a different name to use the same reference solution, use the refdir argument to override the last part of the directory path::

test1 = Test('neumann', run, ['--argument'], refdir='other')
# Giving the ref dir REGRESSION_REF/devtests/mechanics.ring.run/other

You will probably want to disable automatic reference generation for all but one of the tests sharing a directory, to be sure which test the calculated reference is from::

test1.disable_reference_generation()

A key application of this is for running a number of similar tests, for example the same simulation with different numbers of CPUs::

__tests__ = []
for np in [1, 2, 4, 8, 16]:
    argv = ['--experiment', 'neumann', '--np', np]
    test = Test('neumann_np{0}'.format(np), run, argv, refdir='neumann')
    if np != 1:
        # Only generate reference for --np 1
        test.disable_reference_generation()
    __tests__.append(test)

Nightly Summary

Follow the :ref:link <regression-testing-results> to see the results of the nightly runs. Follow the regression-testing-results to see the results of the nightly runs.

Installation From Source

.. _mac-osx-source-install:

Mac OSX Source Install

While CME software has been successfully installed on a growing number of Apple systems, the installation process has not been streamlined or finalized. The current documentation may therefore not be complete or apply to all systems. Comments, questions, or suggestions for improvements are well appreciated by |GPlank| and |KGillette|.

System Requirements

First, installs of both homebrew and Xcode with command line tools are required.

  • homebrew:

.. code-block:: bash

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Xcode: download from App Store. If working with an older OSX, Xcode can be downloaded from apple developer <https://developer.apple.com/xcode/>_.

.. code-block:: bash

 xcode-select --install

Now using homebrew install gcc and wget:

.. code-block:: bash

brew install gcc brew install wget brew install libpng

PETSc and |carp|

.. warning:: Doing a source installation of |carp| requires additional permissions. Please contact |KGillette| before proceeding.

Decide where you are going to do the install of |carp| and PETSc that requires multiple directories and packages. For example: $HOME/install

Set your PETSc environmental variables in your ~/.bashrc. (If necessary, create the .bashrc file):

.. code-block:: bash

export PETSC_DIR=$HOME/install/petsc export PETSC_ARCH=osx-clang-debug2

Update your shell variables by typing source ~/.bashrc. Now download PETSc:

.. code-block:: bash

cd $HOME/install git clone -b maint https://bitbucket.org/petsc/petsc petsc cd petsc

Insert :download:osx-clang-debug.py </downloads/osx-clang-debug.py> into the config directory of PETSc and run the configure scripts:

.. code-block:: bash

cp \(HOME/Downloads/osx-clang-debug.py config python config/osx-clang-debug.py make PETSC_DIR=\)PETSC_DIR PETSC_ARCH=\(PETSC_ARCH all make PETSC_DIR=\)PETSC_DIR PETSC_ARCH=$PETSC_ARCH test

Continue with downloading the developer sources for CARP:

.. code-block:: bash

cd $HOME/install svn co —username=username https://carpentry.medunigraz.at/carp-dcse-pt/branches/mechanics carp-dcse-pt cd carp-dcse-pt

Patch some files:

  • Add :download:switches.def </downloads/switches.def> to $HOME/install/carp-dcse-pt |br| Note: you can set the flags to be different if needed for different packages.

  • Add osx_carp_patch <https://carpentry.medunigraz.at/carpentry/osx_carp_patch.tar.gz>_ to $HOME/install/carp-dcse-pt/CARP. |br| Note: Unpackage using tar -xvf osx_carp_patch.tar.gz

Now build CARP:

.. code-block:: bash

make # you will get an error about switches, don’t fret! make svn_co # updates/checks out related repositories make gengetopt make param make redblack

Add the provided paths (gengetopt and param) to your PATH environment variable (~/.bashrc):

.. code-block:: bash

export PATH=\(PATH:\)HOME/install/carp-dcse-pt/gengetopt-2.22.6/src:$HOME/install/carp-dcse-pt/PrM/src

Update your terminal enviroment (source ~/.bashrc) and finish making CARP:

.. code-block:: bash

bash # to update the shell make all

If you get an error for python2:

  • Check to make sure you have a system-wide installation of python2. If the executable is not calling python2 from /usr/bin then create a symbolic link for it (ln -s python python2) or you can also do a source installation which gives you the python2 executable (suggestion from Christoph Augustin)

  • If you do not have a system-wide installation of python2, install a local one from source and make sure the executable is on your path.

|

carputils And Examples

The documentation for carputils installation is located :ref:here <carputils-basic-install>.

.. code-block:: bash

cd $HOME/install git clone https://username@bitbucket.org:/carpusers/carputils.git mkdir carp-examples cd carp-examples git clone https://username@bitbucket.org:carpusers/tutorials.git

Update your bashrc:

.. code-block:: bash

export PYTHONPATH=\(PYTHONPATH:\)HOME/install/carputils:\(HOME/install/carp-examples export TUTORIALS =\)HOME/install/carp-examples/tutorials

Generate a settings files for carputils:

.. code-block:: bash

cd \(HOME/install/carputils/bin ./cusettings\)HOME/.config/carputils/settings.yaml

Go in and edit your settings.yaml file with the proper directories for CARP-CPU and MESHALYZER

.. code-block:: bash

CPU: \(HOME/install/carpentry/bin MESHALYZER_DIR:\)HOME/install/meshalyzer

In addition to the :ref:carputils basic installation <carputils-manual-installation>, following repositories might be beneficial to have too:

.. code-block:: bash

git clone git@bitbucket.org:carpusers/devtests.git
git clone git@bitbucket.org:carpusers/benchmarks.git
git clone git@bitbucket.org:carpusers/pvprocess.git

|

Installing a Local Python Environment

Many HPC systems use older operating systems with out of data Python distributions, so you may find yourself without the required Python 2.7 version. Follow the below instructions to build and install a local Python environment.

Firstly, get the latest Python source from https://www.python.org/downloads/. Get the latest Python 2 version - carputils does not run in Python 3. Then, make a separate directory to contain the installed Python distribution (including the binaries and any extra modules you will install), e.g.:

.. code-block:: bash

cd $HOME
mkdir python-distribution

Extract the Python source tarball and run the configure script, setting the --prefix and --exec-prefix options to the path of the directory created above:

.. code-block:: bash

tar xvf Python-2.7.10.tar.gz
cd Python-2.7.10
./configure --prefix $HOME/python-distribution --exec-prefix $HOME/python-distribution

Then, build and install Python to the specified folder:

.. code-block:: bash

make
make install

The built Python interpreter should now be present in $HOME/python-distribution/bin. To make this interpreter the default when running python on the command line, add the following line to the bottom of the file ~/.bashrc:

.. code-block:: bash

export PATH=$HOME/python-distribution/bin:$PATH

Finally, install the pip Python package manager to facilitate the easier installation of Python modules later. This is done with:

.. code-block:: bash

python -m ensurepip
pip install --upgrade pip # To update to latest version

|