R2915/UniBe9f72c32f6675master
R2915/UniBe
9f72c32f6675master
UniBe
UniBe
README.md
README.md
Pipeline, a wrapper around Singularity to build, run and test scientific pipelines.
Basic usage:
- ./pipeline.py build - create the container.
- ./pipeline.py run - run the pipeline.
- ./pipeline.py test - prepare test input, run the pipeline, validate output.
Commented sample pipeline description:
See pipeline.yaml
Common options:
- -p description.yaml - choose a pipeline description.
- -i container.img - force a particular container filename.
- -f - force rebuilding the container / test files even if they exist.
- --skip-run - for testing, skip the run stage (e.g. to tweak validation scripts).
Requirements:
- Python
- pyyaml
- argparse
requirements.txt can be used for making a virtualenv.
Testing
Use scicore-pipeline-2a.yaml+Pipeline2 for testing a real pipeline.
slurm-submit-script-WITH-containers.sh is a sample SLURM script to run validation.
TODO
- More flexible test process: prepare stage only, for example
- More flexible build stage: allow preprocessing (e.g. bootstrap downloading the Singularity file)
- Integrity testing: checksums for source / test files?
- A lot more format checks and exception handling!
- More consistent output formatting
c4science · Help