Clone of the renku templates. Defining new project templates relevant to the LSMS
Recent Commits
Commit | Author | Details | Committed | ||||
---|---|---|---|---|---|---|---|
95817f914766 | anciaux | Merge branch 'master' of https://github.com/SwissDataScienceCenter/renku… | Jan 11 2023 | ||||
f2a778119cb4 | Rok Roškar/GitHub | chore: fix project config (#155) | Dec 23 2022 | ||||
c83a1b149b3e | Rok Roškar/GitHub | chore: restructure dockerfile for build performance and caching (#154) | Dec 23 2022 | ||||
0fb0b9566bc1 | champost/GitHub | chore : standardize .gitlab-ci.yml across templates (#151) | Nov 7 2022 | ||||
b4a335b6c146 | Tasko Olevski/GitHub | chore: update images to renkulab-docker 0.13.1 (#152) | Oct 26 2022 | ||||
e71d20532928 | anciaux | fix requirements | Oct 25 2022 | ||||
fe4a082ef78c | anciaux | saving state | Oct 25 2022 | ||||
de7e5e1e792c | anciaux | adding deps in requirements | Oct 25 2022 | ||||
d7b251275330 | anciaux | update dockerfile | Oct 25 2022 | ||||
e54ba9a821a7 | anciaux | saving readme | Oct 24 2022 | ||||
2e7fbdcafabf | anciaux | cleaning | Oct 24 2022 | ||||
8116fd9080c1 | anciaux | clean gitignore | Oct 24 2022 | ||||
8c2a988d09af | anciaux | trick for .bd | Oct 24 2022 | ||||
c2b5638c0d31 | anciaux | post-treatment update | Oct 24 2022 | ||||
1918d9bf873c | anciaux | adding bd template | Oct 24 2022 |
README.md
renku-project-template
A repository of base templates for new Renku projects. The next sections outline what different files in the template are used for.
For running interactive environments from Renkulab
Dockerfile - File for building a docker image that you can launch from renkulab, to work on your project in the cloud. Template-supplied contents will allow you to launch an interactive environment from renkulab, with pre-installed renku CLI and software dependencies that you put into your requirements.txt, environment.yml, or install.R. You can and should add to this Dockerfile if libraries you install require linux software installations as well; for more information see: https://github.com/SwissDataScienceCenter/renkulab-docker.
.gitlab-ci.yml - Configuration for running gitlab CI, which builds a docker image out of the project on git push to renkulab so that you can launch your interactive environment (don't remove, but you can modify to add extra CI functionality).
.dockerignore - Files and directories to be excluded from docker build (you can
append to this list); https://docs.docker.com/engine/reference/builder/#dockerignore-file.
Setting the version of the renku-cli
The default version of the renku CLI used in the interactive environment is specified in the Dockerfile in a line similar to this:
ARG RENKU_VERSION={{ __renku_version__ | default("0.16.0") }}
The client creating the project (either via the UI in RenkuLab or the renku CLI) can override this default setting. The version is set as follows:
- if the client (the renku core service or the renku CLI) is using a released version, then pass this to the project template
- if the client is on a development version, use the default provided by the template
For managing software dependencies
requirements.txt - Required by template's Dockerfile; add your python pip dependencies here.
environment.yml - Required by template's Dockerfile; add your python conda dependencies here.
install.R - Required by template's Dockerfile (for r-based projects only).
For the landing page for your project
README.md - Edit this file to provide information about your own project. Initial contents explain how to use a renku project.
For renku CLI
.renku - Directory containing renku metadata that renku commands update (caution: don't update this manually).
.renkulfsignore - File similar to .gitignore for telling renku to NOT store listed files in git LFS. Use in conjunction with renku config lfs_threshold <[size]kb> to tell renku to NOT store files above a threshold size in LFS. Initial configuration is set to 100kb.
By default, renku commands (like renku run and renku dataset) store all output files above a configurable threshold size of 100KB in git LFS to prevent accidentally committing large files to git. It's bad to git commit large files (e.g. datasets, graphics, videos, audio samples) without being tracked by git LFS, because they slow down git commands (and thus renku commands). However, sometimes:
- an imported dataset will come with markdown (*.md) and/or code (e.g. *.py).
- a code file (like *.ipynb) will be generated from a renku run (e.g. with papermill).
- generated or imported data could be small (e.g. <100kb)
Tracking files with LFS is good, but limits your ability to use commands like git diff to view changes, and to see the contents of the files in the project's page on renkulab.
Thus, you can edit .renkulfsignore to add files with particular paths or extensions that are relevant for your project. renku commands will consult .renkulfsignore and not track those files with git LFS.
Note: When you start a new interactive environment, by default the LFS-tracked files (e.g. files above the configured threshold AND not on this list) are in their "pointer" form. Run renku storage pull <filepath> to pull the real content into each file, or git lfs pull to replace all pointers with real content all at once. Since these are large files, you might be better off pulling them one at a time.
For organizing project files
data - Initially empty directory where renku dataset creates subdirectories for your named datasets and the files you add to those datasets (if you haven't or will not be creating renku datasets, you can remove this directory).
notebooks - Initially empty directory to help you organize jupyter notebooks (not a requirement, you can remove this directory).
For git to ignore
.gitignore - Files and directories to be excluded from git repository (this
template only requires the #renku section, but the others are nice-to-haves for common paths to ignore).