Skip to content

Latest commit

 

History

History
104 lines (72 loc) · 7.13 KB

CONTRIBUTING.md

File metadata and controls

104 lines (72 loc) · 7.13 KB
Raw

Contributing

This document is the attempt to collect some rough rules for tools to follow in this repository, to facilitate their consistency and interoperability. This document is an extension to the Naming and Annotation Conventions for Tools in the Image Community in Galaxy and compatibility should be maintained. This document is work in progress.

How to Contribute:

Terminology

Label maps are images with pixel-level annotations, usually corresponding to distinct image regions (e.g., objects). We avoid the terms label image and labeled image, since these can be easily confused with image-level annotations (instead of pixel-level). The labels (pixel values) must uniquely identify the labeled image regions (i.e. labels must be unique, even for non-adjacent image regions). If a label semantically corresponds to the image background, that label should be 0.

Binary images are a special case of label maps with only two labels (e.g., image background and image foreground). To facilitate visual perception, the foreground label should correspond to white (value 255 for uint8 images and value 65535 for uint16 images), since background corresponds to the label 0, which is black.

Intensity images are images which are not label maps (and thus neither binary images).

File types

In tool wrappers which use a Python script, image loading should be performed by using the giatools package (see #119). If such wrappers only support single-channel 2-D images, the structure of the input should be verified right after loading the image:

im = giatools.io.imread(args.input)
im = np.squeeze(im)  # remove axes with length 1
assert im.ndim == 2

Tools with label map inputs should accept PNG and TIFF files. Tools with label map outputs should produce either uint16 single-channel PNG or uint16 single-channel TIFF. Using uint8 instead of uint16 is also acceptable, if there definetely are no more than 256 different labels. Using uint8 should be preferred for binary images.

Note

It is a common misconception that PNG files must be RGB or RGBA, and that only uint8 pixel values are supported. For example, the cv2 module (OpenCV) can be used to create single-channel PNG files, or PNG files with uint16 pixel values. Such files can then be read by giatools.io.imread or skimage.io.imread without issues (however, skimage.io.imwrite seems not to be able to write such PNG files).

Tools with intensity image inputs should accept PNG and TIFF files. Tools with intensity image outputs can be any data type and either PNG or TIFF. Image outputs meant for visualization (e.g., segmentation overlays, charts) should be PNG.

Testing

Testing infrastructure

The support for the new image_diff output verification method and assertions for image data for Galaxy tool testing probably won't be available in Galaxy before 24.1 is released.

Meanwhile, they are already available in the CI of the galaxy-image-analyis repostiroy! 🎉 #117

To also use them locally, you need to install the development versions of two Galaxy packages, pillow, and tifffile:

python -m pip install git+https://git@github.com/kostrykin/galaxy.git@galaxy-image-analysis#subdirectory=packages/util
python -m pip install git+https://git@github.com/kostrykin/galaxy.git@galaxy-image-analysis#subdirectory=packages/tool_util
python -m pip install pillow tifffile

The galaxy-image-analysis branch of the https://github.com/kostrykin/galaxy fork is the same as the 23.1 release of Galaxy, plus the support for the image-based verification extensions.

In addition, instead of running planemo test, you should use:

planemo test --galaxy_source https://github.com/kostrykin/galaxy --galaxy_branch galaxy-image-analysis

Linting with planemo lint works as usual.

Writing tests

We recommend using macros for verification of image outputs. The macros are loaded as follows:

<macros>
    <import>tests.xml</import>
</macros>

Testing binary image outputs

For testing of binary image outputs we recommend using the mae metric (mean absolute error). The default value for eps of 0.01 is rather strict, and for 0/1 binary images this asserts that at most 1% of the image pixels are labeled differently:

<expand macro="tests/binary_image_diff" name="output" value="output.tif" ftype="tiff"/>

For 0/255 binary images, the same 1% tolerance would be achieved by increasing eps to 2.25. The macro also ensures that the image contains two distinct label values.

Testing label map outputs

For testing of non-binary label map outputs with interchangeable labels, we recommend using the iou metric (one minus the intersection over the union). With the default value of eps of 0.01, this asserts that there is no labeled image region with an intersection over the union of less than 99%:

<expand macro="tests/label_image_diff" name="output" value="output.tif" ftype="tiff"/>

Label 0 is commonly connotated as the image background, and is not interchangable by default. Use pin_labels="" to make it interchangable.

Testing intensity image outputs

For testing of intensity image outputs we recommend the rms metric (root mean square), because it is very sensitive to large pixel value differences, but tolerates smaller differences:

<expand macro="tests/intensity_image_diff" name="output" value="output.tif" ftype="tiff"/>

For uint8 and uint16 images, increasing the default value of eps to 1.0 should be tolerable, if required.

Future extensions

Below is a list of open questions:

  • How do we want to cope with multi-channel label maps? For example, do or can we distinguish RGB labels from multi-channel binary masks, which are sometimes used to represent overlapping objects?

  • How can we distinguish multi-channel 2-D images from single-channel 3-D images?

  • How can we make clear to the user, whether a tool requires a 2-D image or also supports 3-D?