Using the OpenSlide test suite

Setting up your system

To run the test suite, you will need:

Sanitize mode requires:

Valgrind mode requires:

Coverage mode requires:

Running test cases

Run all test cases with:

./driver run

To run a subset of the test cases, you can use a glob pattern:

./driver run 'mirax*'

You can also run the test cases under Clang sanitizers. This will catch undefined behavior, memory leaks, invalid memory accesses, double frees, etc., but is somewhat slow.

./driver sanitize

You can also run the test cases under Valgrind. This will catch memory leaks, invalid memory accesses, double frees, etc. It is also very slow.

./driver valgrind

To generate a test coverage report:

./driver coverage out.txt

To report openslide_open() times for primary test cases:

./driver time

To generate a test render of regions from various slides:

./driver mosaic out.png

Selecting a cache directory

Cached slide data is stored in ./_slidedata by default. To use a different directory, set the OPENSLIDE_TEST_CACHE environment variable to the desired directory path.

Expected failures

On some platforms, certain test cases may fail due to problems in other libraries. To prevent those cases from cluttering the test report, you can set the OPENSLIDE_TEST_XFAIL and/or OPENSLIDE_VALGRIND_XFAIL environment variables to a comma-separated list of cases which are expected to fail during run or valgrind, respectively.

Using a local mirror

If you have a local mirror of the openslide-testdata repository, you can specify its URL using the OPENSLIDE_TESTDATA_URL environment variable.

Adding new slides

This should only be necessary when adding new slide formats or variants.

Select the smallest appropriate slide file from openslide-testdata and add it, and its SHA-256 hash, to test/cases/slides.yaml. Then create a successful test case including these declarations:

# This is the "vanilla" test case for the slide
primary: true
# Check properties
properties:
  openslide.vendor: <vendor>
  # Detect inadvertent changes to the quickhash definition
  openslide.quickhash-1: <value>

You should also add a new mosaic region.

Creating new test cases

First, choose a slide and create a skeleton test case for it:

./driver create Mirax/CMU-1.zip name-of-test-case
cd cases/name-of-test-case

Edit the slide file(s):

hexedit slide/CMU-1/Index.dat

Test that your edits produced the desired results:

../../try_open slide/CMU-1.mrxs

Edit the test metadata:

vi config.yaml

If the slide should open successfully, change success to true. Otherwise, set error to a regular expression matching the error string that should be returned by openslide_get_error() after openslide_open() returns.

If openslide_detect_vendor() should return a different result than for the original slide (e.g., if OpenSlide falls back to generic-tiff), change vendor to the new string or null for NULL.

If the test should only be run if particular OpenSlide dependencies are available, set requires to a list of feature flags. Currently the supported feature flags are:

tiff-log-callbacks
OpenSlide can capture warning/error log messages from libtiff

Pack the test:

cd ../..
./driver pack name-of-test-case

This will generate binary diffs of the modified slide file(s).

Finally, commit files to Git:

Annotated config.yaml

# The openslide-testdata file containing the slide, and the relative path
# within that file (if a Zip).  These are set by "./driver create" and you
# shouldn't need to change them.
base: Hamamatsu-vms/CMU-1.zip
slide: CMU-1-40x - 2010-01-12 13.24.05.vms

# true if openslide_open() and any openslide_read_region() calls should
# succeed, false otherwise.
success: false

# If success is false, a regex for the expected openslide_get_error()
# string.  Should be anchored when possible.  May need to be quoted.
error: ^Tile size not consistent$

# The string that should be returned by openslide_detect_vendor(),
# or null.
vendor: hamamatsu

# True if this is the "vanilla" (unmodified) test case for this slide.
# If true, extended tests will be run, and the case will be evaluated
# during "./driver time".  Defaults to false.
primary: false

# Optional list of features required by this test.  If a specified
# feature is not available, the test will be skipped.
requires: [some-library]

# Optional list of debug options to include in the OPENSLIDE_DEBUG
# environment variable.
debug:
  - jpeg-markers

# Optional list of regions to read.  Arguments are in
# openslide_read_region() order: [x, y, level, w, h].
regions:
  - [0, 0, 1, 100, 100]
  - [1000, 1000, 2, 1, 1]

# Optional map of property values to check against the properties returned
# by OpenSlide.
properties:
  openslide.vendor: hamamatsu

# Generate test case files programmatically at unpack time.  Useful when
# test cases can be deterministically produced by a program but their deltas
# are too large to check in.  Of course this introduces a dependency on the
# specified program.  Use with caution, and only as a last resort.
generate:
  ? "CMU-1-40x - 2010-01-12 13.24.05(1,1).jpg"
  : "jpegtran -restart 256B -outfile %(out)s %(in)s"

# Copy test case files during unpack.
copy:
  ? DCM_0_copy.dcm
  : DCM_0.dcm

# Rename or move test case files during unpack.
rename:
  ? "CMU-1-40x - 2010-01-12 13.24.05(1,1).jpg"
  : "subdir/CMU-1-40x - 2010-01-12 13.24.05(1,1).jpg"

Tips

Adding mosaic regions

Mosaic regions are specified in test/cases/mosaic.ini. Regions are rendered in order and labeled with INI section names:

[MIRAX Exported]
base = Mirax/CMU-1-Exported.zip
; Omit for single-file slides
slide = CMU-1-Exported.mrxs
x = 58240
y = 171905

Mosaic regions are always level 0 and 256 x 256 pixels. Find an appropriate region by testing with slidetool region read and OPENSLIDE_DEBUG=tiles. A good region includes a tile boundary.