README.md
9f72c32f6675master

README.md
View Options

			# Pipeline, a wrapper around Singularity to build, run and test scientific pipelines.

			## Basic usage:

			* `./pipeline.py build` - create the container.
			* `./pipeline.py run` - run the pipeline.
			* `./pipeline.py test` - prepare test input, run the pipeline, validate output.

			## Commented sample pipeline description:

			See `pipeline.yaml`

			## Common options:

			* `-p description.yaml` - choose a pipeline description.
			* `-i container.img` - force a particular container filename.
			* `-f` - force rebuilding the container / test files even if they exist.
			* `--skip-run` - for testing, skip the run stage (e.g. to tweak validation scripts).

			## Requirements:

			* Python
			* pyyaml
			* argparse

			`requirements.txt` can be used for making a virtualenv.

			## Testing

			Use `scicore-pipeline-2a.yaml`+`Pipeline2` for testing a real pipeline.

			`slurm-submit-script-WITH-containers.sh` is a sample SLURM script to run validation.

			## TODO

			* More flexible test process: prepare stage only, for example
			* More flexible build stage: allow preprocessing (e.g. bootstrap downloading the Singularity file)
			* Integrity testing: checksums for source / test files?
			* A lot more format checks and exception handling!
			* More consistent output formatting