Pipeline, a wrapper around Singularity to build, run and test scientific pipelines.

Basic usage:

• ./pipeline.py build - create the container.
• ./pipeline.py run - run the pipeline.
• ./pipeline.py test - prepare test input, run the pipeline, validate output.

Commented sample pipeline description:

See pipeline.yaml

Common options:

• -p description.yaml - choose a pipeline description.
• -i container.img - force a particular container filename.
• -f - force rebuilding the container / test files even if they exist.
• --skip-run - for testing, skip the run stage (e.g. to tweak validation scripts).

Requirements:

• Python
  • pyyaml
  • argparse

requirements.txt can be used for making a virtualenv.

Testing

Use scicore-pipeline-2a.yaml+Pipeline2 for testing a real pipeline.

slurm-submit-script-WITH-containers.sh is a sample SLURM script to run validation.

TODO

• More flexible test process: prepare stage only, for example
• More flexible build stage: allow preprocessing (e.g. bootstrap downloading the Singularity file)
• docker2singularity build type?
• Integrity testing: checksums for source / test files?
• A lot more format checks and exception handling!
• More consistent output formatting