{ "cells": [ { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "slide" } }, "source": [ "# So, you want to go open Science ?\n", "## Version, colaborate, share, test, document !\n", "\n", "
\n", "\n", "We will manly show you how to do things, and give you the full code of what we did\n", "\n", "
\n", "\n", "- https://c4science.ch/source/so-workshop/\n", "- https://c4science.ch/source/so-workshop-test/\n", "- https://c4science.ch/source/so-slides/\n", "\n", "\n", "


\n", "SCITAS training with expertise from the EPFL Library
\n", "Nicolas Richart and Jean-Baptiste Aubort - SCITAS" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "slide" } }, "source": [ "# First thing first : Version your stuffs" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "source": [ "# Create a repository\n", "![](images/c4s-repository.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "source": [ "## Some basics on git\n", "- Let this represent a repository\n", "![](images/git.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "source": [ "- Clone an existing repository\n", "![](images/git-clone.png)" ] }, { "cell_type": "code", "execution_count": 89, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "cd ..\n", "rm -rf so-workshop" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cloning into 'so-workshop'...\n", "remote: Counting objects: 191, done.\u001b[K\n", "remote: Compressing objects: 100% (181/181), done.\u001b[K\n", "remote: Total 191 (delta 89), reused 0 (delta 0)\u001b[K\n", "Receiving objects: 100% (191/191), 17.59 KiB | 2.20 MiB/s, done.\n", "Resolving deltas: 100% (89/89), done.\n" ] } ], "source": [ "git clone https://c4science.ch/source/so-workshop.git\n", "cd so-workshop" ] }, { "cell_type": "code", "execution_count": 91, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "echo \"Add a modification to the readme\" >> README" ] }, { "cell_type": "code", "execution_count": 92, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "On branch master\n", "Your branch is up to date with 'origin/master'.\n", "\n", "Changes not staged for commit:\n", " (use \"git add ...\" to update what will be committed)\n", " (use \"git checkout -- ...\" to discard changes in working directory)\n", "\n", "\t\u001b[31mmodified: README\u001b[m\n", "\n", "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n" ] } ], "source": [ "# edit README\n", "git status" ] }, { "cell_type": "code", "execution_count": 93, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "git add README" ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "On branch master\n", "Your branch is up to date with 'origin/master'.\n", "\n", "Changes to be committed:\n", " (use \"git reset HEAD ...\" to unstage)\n", "\n", "\t\u001b[32mmodified: README\u001b[m\n", "\n" ] } ], "source": [ "git status" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[master 8971788] Added a change to README\n", " 1 file changed, 1 insertion(+)\n" ] } ], "source": [ "git commit -m \"Added a change to README\"" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "* \u001b[33mcommit 89717881e7241f06a6731bd4211322bf7da72935\u001b[m\u001b[33m (\u001b[m\u001b[1;36mHEAD -> \u001b[m\u001b[1;32mmaster\u001b[m\u001b[33m)\u001b[m\u001b[m\n", "\u001b[31m|\u001b[m Author: Nicolas Richart \u001b[m\n", "\u001b[31m|\u001b[m Date: Mon Dec 3 04:32:43 2018 +0100\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m Added a change to README\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "* \u001b[33mcommit 59738410b9ec50d0a20cbdc5639e045084bbc5a1\u001b[m\u001b[33m (\u001b[m\u001b[1;31morigin/master\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/HEAD\u001b[m\u001b[33m)\u001b[m\u001b[m\n", "\u001b[31m|\u001b[m Author: Jean-Baptiste Aubort \u001b[m\n", "\u001b[31m|\u001b[m Date: Fri Nov 30 16:52:44 2018 +0100\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m Fix branch name\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "* \u001b[33mcommit b896907f7b01e9490c47f85f57783cb36040c39d\u001b[m\u001b[m\n", "\u001b[31m|\u001b[m Author: Jean-Baptiste Aubort \u001b[m\n", "\u001b[31m|\u001b[m Date: Fri Nov 30 16:51:12 2018 +0100\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m Update branch names\u001b[m\n", "\u001b[K\u001b[?1l\u001b>" ] } ], "source": [ "git log --graph -n3" ] }, { "cell_type": "code", "execution_count": 97, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "skip" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Enumerating objects: 36, done.\n", "Counting objects: 100% (36/36), done.\n", "Delta compression using up to 4 threads\n", "Compressing objects: 100% (23/23), done.\n", "Writing objects: 100% (36/36), 3.68 KiB | 3.68 MiB/s, done.\n", "Total 36 (delta 6), reused 28 (delta 5)\n", "To ssh://c4science.ch/source/so-workshop-test.git\n", " + 1294907...8971788 master -> master (forced update)\n", "Branch 'master' set up to track remote branch 'master' from 'origin'.\n" ] } ], "source": [ "# In order to push without changing the repository origin\n", "git remote set-url origin ssh://git@c4science.ch/source/so-workshop-test.git\n", "git push --force -u origin master" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "source": [ "- `git commit`\n", "![](images/git-commit.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "source": [ "- often someone else already commited\n", "![](images/git-commit-remote.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "source": [ "- `git pull`\n", "![](images/git-pull.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "source": [ "- `git push`\n", "![](images/git-push.png)" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "slide" } }, "source": [ "# Collaborate through projects" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "source": [ "## Why projects\n", "\n", "- To tidy your repositories\n", "- To handle permissions\n", "- To define tasks\n", "- To attach information" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "slide" } }, "source": [ "# Now that your are all setup : add a workflow" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "-" } }, "source": [ "## Why a workflow\n", "\n", "- Helps to separate development\n", "- Helps to release a code\n", "- Helps to review to code\n", "- Helps to maintain different versions" ] }, { "cell_type": "markdown", "metadata": { "ein.tags": "worksheet-0", "slideshow": { "slide_type": "subslide" } }, "source": [ "## Minimalist workflow\n", "\n", "- Feature branches" ] }, { "cell_type": "code", "execution_count": 98, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to a new branch 'feature/workflow'\n" ] } ], "source": [ "git checkout -b feature/workflow" ] }, { "cell_type": "code", "execution_count": 99, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[feature/workflow cd9f182] Added a new feature to demonstrate a simple workflow\n", " 1 file changed, 1 insertion(+)\n", " create mode 100644 new_feature\n" ] } ], "source": [ "echo \"Adding my feature\" >> new_feature\n", "git add new_feature\n", "git commit -m \"Added a new feature to demonstrate a simple workflow\"" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- Merge your changes back in master" ] }, { "cell_type": "code", "execution_count": 100, "metadata": { "autoscroll": false, "ein.hycell": false, "ein.tags": "worksheet-0", "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'master'\n", "Your branch is up to date with 'origin/master'.\n", "Merge made by the 'recursive' strategy.\n", " new_feature | 1 \u001b[32m+\u001b[m\n", " 1 file changed, 1 insertion(+)\n", " create mode 100644 new_feature\n" ] } ], "source": [ "git checkout master\n", "git merge --no-ff --no-edit feature/workflow" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "* \u001b[33mcommit 19f5c5ca82998d3259cd80df70e770cbf07c56f9\u001b[m\u001b[33m (\u001b[m\u001b[1;36mHEAD -> \u001b[m\u001b[1;32mmaster\u001b[m\u001b[33m)\u001b[m\u001b[m\n", "\u001b[31m|\u001b[m\u001b[32m\\\u001b[m Merge: 8971788 cd9f182\u001b[m\n", "\u001b[31m|\u001b[m \u001b[32m|\u001b[m Author: Nicolas Richart \u001b[m\n", "\u001b[31m|\u001b[m \u001b[32m|\u001b[m Date: Mon Dec 3 04:32:47 2018 +0100\u001b[m\n", "\u001b[31m|\u001b[m \u001b[32m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m \u001b[32m|\u001b[m Merge branch 'feature/workflow'\u001b[m\n", "\u001b[31m|\u001b[m \u001b[32m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m * \u001b[33mcommit cd9f182ab178f62d1cfb8a1dee43e3bc80ca11fb\u001b[m\u001b[33m (\u001b[m\u001b[1;32mfeature/workflow\u001b[m\u001b[33m)\u001b[m\u001b[m\n", "\u001b[31m|\u001b[m\u001b[31m/\u001b[m Author: Nicolas Richart \u001b[m\n", "\u001b[31m|\u001b[m Date: Mon Dec 3 04:32:47 2018 +0100\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "\u001b[31m|\u001b[m Added a new feature to demonstrate a simple workflow\u001b[m\n", "\u001b[31m|\u001b[m \u001b[m\n", "* \u001b[33mcommit 89717881e7241f06a6731bd4211322bf7da72935\u001b[m\u001b[33m (\u001b[m\u001b[1;31morigin/master\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/HEAD\u001b[m\u001b[33m)\u001b[m\u001b[m\n", "\u001b[32m|\u001b[m Author: Nicolas Richart \u001b[m\n", "\u001b[32m|\u001b[m Date: Mon Dec 3 04:32:43 2018 +0100\u001b[m\n", "\u001b[32m|\u001b[m \u001b[m\n", "\u001b[32m|\u001b[m Added a change to README\u001b[m\n", "\u001b[K\u001b[?1l\u001b>" ] } ], "source": [ "git log --graph -n3" ] }, { "cell_type": "code", "execution_count": 102, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Already on 'master'\n", "Your branch is ahead of 'origin/master' by 2 commits.\n", " (use \"git push\" to publish your local commits)\n", "Enumerating objects: 5, done.\n", "Counting objects: 100% (5/5), done.\n", "Delta compression using up to 4 threads\n", "Compressing objects: 100% (3/3), done.\n", "Writing objects: 100% (4/4), 413 bytes | 413.00 KiB/s, done.\n", "Total 4 (delta 2), reused 0 (delta 0)\n", "To ssh://c4science.ch/source/so-workshop-test.git\n", " 8971788..19f5c5c master -> master\n" ] } ], "source": [ "git checkout master \n", "git push" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ " - Review the code using `arcanist`" ] }, { "cell_type": "code", "execution_count": 103, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to a new branch 'feature/review'\n", "[feature/review a00f49c] Added a new feature to demonstrate a review\n", " 1 file changed, 1 insertion(+)\n", "[feature/review 71efe12] Adding a file\n", " 1 file changed, 0 insertions(+), 0 deletions(-)\n", " create mode 100644 new_file\n" ] } ], "source": [ "git checkout -b feature/review\n", "echo \"Adding my feature that needs review\" >> new_feature\n", "git add new_feature\n", "git commit -m \"Added a new feature to demonstrate a review\"\n", "touch new_file\n", "git add new_file\n", "git commit -m \"Adding a file\"" ] }, { "cell_type": "code", "execution_count": 104, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "cat > ../.arcmessage << EOF\n", "Adding a reviewed feature\n", "Test Plan: none yet\n", "Reviewers: nrichart\n", "EOF\n", "\n" ] }, { "cell_type": "code", "execution_count": 105, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Linting...\n", "No lint engine configured for this project.\n", "Running unit tests...\n", "No unit test engine is configured for this project.\n", "\u001b[44m\u001b[1m SKIP STAGING \u001b[m\u001b[49m No staging area is configured for this repository.\n", "Created a new Differential revision:\n", " \u001b[1mRevision URI:\u001b[m \u001b[4mhttps://c4science.ch/D220\u001b[m\n", "\n", "Included changes:\n", " M new_feature\n", " A new_file\n" ] } ], "source": [ "arc diff master -F ../.arcmessage --no-amend" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![](images/c4s-differencial.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "- Once the review accepted it can be landed" ] }, { "cell_type": "code", "execution_count": 108, "metadata": { "scrolled": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Landing current branch 'feature/review'.\n", "\u001b[44m\u001b[1m TARGET \u001b[m\u001b[49m Landing onto \"master\", the default target under git.\n", "\u001b[44m\u001b[1m REMOTE \u001b[m\u001b[49m Using remote \"origin\", the default remote under git.\n", "\u001b[44m\u001b[1m FETCH \u001b[m\u001b[49m Fetching origin/master...\n", "These commits will be landed:\n", "\n", " - 71efe12 Adding a file\n", " - a00f49c Added a new feature to demonstrate a review\n", "\n", "Landing revision 'D220: Adding a reviewed feature'...\n", "\u001b[1m\u001b[42m BUILDS PASSED \u001b[49m\u001b[m Harbormaster builds for the active diff completed successfully.\n", "\u001b[44m\u001b[1m PUSHING \u001b[m\u001b[49m Pushing changes to \"origin/master\".\n", "Enumerating objects: 9, done.\n", "Counting objects: 100% (9/9), done.\n", "Delta compression using up to 4 threads\n", "Compressing objects: 100% (6/6), done.\n", "Writing objects: 100% (7/7), 677 bytes | 225.00 KiB/s, done.\n", "Total 7 (delta 4), reused 0 (delta 0)\n", "To ssh://c4science.ch/source/so-workshop-test.git\n", " 19f5c5c..52ec4a6 52ec4a6c006cd9b0b95957dc810d98bc2cc84139 -> master\n", "\u001b[44m\u001b[1m UPDATE \u001b[m\u001b[49m Local \"master\" tracks target remote \"origin/master\", checking out and pulling changes.\n", "\u001b[44m\u001b[1m PULL \u001b[m\u001b[49m Checking out and pulling \"master\".\n", "Cleaning up branch \"feature/review\"...\n", "(Use `git checkout -b feature/review 71efe12719648965e96ad146412d0c3726dc79a0` if you want it back.)\n", "\u001b[42m\u001b[1m DONE \u001b[m\u001b[49m Landed changes.\n" ] } ], "source": [ "arc land --merge --revision D220" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Adding a public verison of the code" ] }, { "cell_type": "code", "execution_count": 117, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "warning: no common commits\n", "remote: Enumerating objects: 3, done.\u001b[K\n", "remote: Counting objects: 100% (3/3), done.\u001b[K\n", "remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0\u001b[K\n", "Unpacking objects: 100% (3/3), done.\n", "From github.com:nrichart/so-workshop\n", " * [new branch] master -> public/master\n" ] } ], "source": [ "git remote add public git@github.com:nrichart/so-workshop.git\n", "git fetch public" ] }, { "cell_type": "code", "execution_count": 119, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Branch 'public' set up to track remote branch 'master' from 'public'.\n", "Switched to a new branch 'public'\n" ] } ], "source": [ "git checkout -b public public/master" ] }, { "cell_type": "code", "execution_count": 122, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Squash commit -- not updating HEAD\n", "Automatic merge went well; stopped before committing as requested\n" ] } ], "source": [ "git merge --squash --allow-unrelated-histories master" ] }, { "cell_type": "code", "execution_count": 125, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "On branch public\n", "Your branch is ahead of 'public/master' by 1 commit.\n", " (use \"git push\" to publish your local commits)\n", "\n", "nothing to commit, working tree clean\n", "Enumerating objects: 9, done.\n", "Counting objects: 100% (9/9), done.\n", "Delta compression using up to 4 threads\n", "Compressing objects: 100% (5/5), done.\n", "Writing objects: 100% (8/8), 885 bytes | 885.00 KiB/s, done.\n", "Total 8 (delta 0), reused 2 (delta 0)\n", "To github.com:nrichart/so-workshop.git\n", " cb63ebe..01b2301 public -> master\n", "Branch 'public' set up to track remote branch 'master' from 'public'.\n" ] } ], "source": [ "git commit -m \"First public revision\"\n", "git push -u public public:master" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Continuous Integration (CI)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "## What is CI ?\n", "\n", "- Automation of build and tests\n", "- Run tests at each commit\n", "- You need to write tests !" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "## Why use CI ?\n", "\n", "- Detect bugs when they are introduced\n", "- Detect when fixed bugs appear again (regression)\n", "- Publish code that works" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Ensure code has a certain quality level\n", "- Linters\n", "- Test Coverage" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Jenkins on c4science" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## https://c4science.ch/jobs\n", "![](images/c4s-jenkinsjob.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## https://jenkins.c4science.ch/\n", "![](images/c4s-jenkins.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Example\n", "## Jenkins base" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "cd ../so-workshop" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'feature/jenkins-base'\n", "Your branch is up to date with 'origin/feature/jenkins-base'.\n", "\u001b[01;34m.\u001b[00m\n", "├── \u001b[01;34mcode\u001b[00m\n", "│   ├── mylib.py\n", "│   └── \u001b[01;32mtest.py\u001b[00m\n", "├── Jenkinsfile\n", "└── README\n", "\n", "1 directory, 4 files\n" ] } ], "source": [ "git checkout feature/jenkins-base\n", "tree" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#!/usr/bin/env python3\n", "\n", "class MyLib():\n", "\n", " def __init__(self):\n", " pass\n", "\n", " def MyFunc(self):\n", " return 'test'\n", "\n", "if __name__ == '__main__':\n", " pass\n" ] } ], "source": [ "cat code/mylib.py" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#!/usr/bin/env python3\n", "import unittest, pytest\n", "import mylib\n", "\n", "class Test(unittest.TestCase):\n", "\n", " def test_return(self):\n", " m = mylib.MyLib()\n", " self.assertTrue(m.MyFunc() == 'test')\n", "\n", "if __name__ == '__main__':\n", " pytest.main(['./test.py'])\n" ] } ], "source": [ "cat code/test.py" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pipeline {\n", "\n", " agent {\n", " docker {\n", " image 'python:3.7'\n", " }\n", " }\n", "\n", " environment {\n", " HOME = \"$WORKSPACE\"\n", " }\n", "\n", " stages {\n", " stage('install dependencies') {\n", " steps {\n", " sh 'pip3 install --user pytest'\n", " }\n", " }\n", " stage('run tests') {\n", " steps {\n", " sh '$HOME/.local/bin/pytest code/test.py'\n", " }\n", " }\n", " }\n", "\n", "}\n" ] } ], "source": [ "cat Jenkinsfile" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![](images/jenkins-base.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Example\n", "## Jenkins dockerfile" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'feature/jenkins-dockerfile'\n", "Your branch is up to date with 'origin/feature/jenkins-dockerfile'.\n", "\u001b[01;34m.\u001b[00m\n", "├── \u001b[01;34mcode\u001b[00m\n", "│   ├── mylib.py\n", "│   └── \u001b[01;32mtest.py\u001b[00m\n", "├── Dockerfile\n", "├── Jenkinsfile\n", "└── README\n", "\n", "1 directory, 5 files\n" ] } ], "source": [ "git checkout feature/jenkins-dockerfile\n", "tree" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "FROM python:3.7\n", "RUN pip3 install --no-cache-dir pytest\n" ] } ], "source": [ "cat Dockerfile" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pipeline {\n", "\n", " agent {\n", " dockerfile true \n", " }\n", "\n", " stages {\n", " stage('run tests') {\n", " steps {\n", " sh 'pytest code/test.py'\n", " }\n", " }\n", " }\n", "\n", "}\n" ] } ], "source": [ "cat Jenkinsfile" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Example\n", "### Jenkins junit" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'feature/jenkins-junit'\n", "Your branch is up to date with 'origin/feature/jenkins-junit'.\n", "\u001b[01;34m.\u001b[00m\n", "├── \u001b[01;34mcode\u001b[00m\n", "│   ├── mylib.py\n", "│   └── \u001b[01;32mtest.py\u001b[00m\n", "├── Dockerfile\n", "├── Jenkinsfile\n", "└── README\n", "\n", "1 directory, 5 files\n" ] } ], "source": [ "git checkout feature/jenkins-junit\n", "tree" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pipeline {\n", "\n", " agent {\n", " dockerfile true \n", " }\n", "\n", " stages {\n", " stage('run tests') {\n", " steps {\n", " sh 'pytest --junitxml results.xml code/test.py'\n", " }\n", " }\n", " }\n", "\n", " post {\n", " always {\n", " junit 'results.xml'\n", " }\n", " }\n", "\n", "}\n" ] } ], "source": [ "cat Jenkinsfile" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![](images/jenkins-junit.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![](images/jenkins-junit2.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# C4science\n", "## Arcanist\n", "Command line tool to interact with c4science" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### Linters\n", "\n", "> A linter or lint refers to tools that analyze source code to flag programming errors, bugs, stylistic errors, and suspicious constructs.\n", "\n", "- https://en.wikipedia.org/wiki/Lint_(software)\n", "- https://c4science.ch/w/c4science/lint/" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'feature/arcanist-linter'\n", "Your branch is up to date with 'origin/feature/arcanist-linter'.\n", "\u001b[01;34m.\u001b[00m\n", "├── .arclint\n", "├── \u001b[01;34mcode\u001b[00m\n", "│   ├── mylib.py\n", "│   └── \u001b[01;32mtest.py\u001b[00m\n", "├── Dockerfile\n", "├── .gitignore\n", "├── Jenkinsfile\n", "└── README\n", "\n", "1 directory, 7 files\n" ] } ], "source": [ "git checkout feature/arcanist-linter\n", "tree -a -I \".git\"" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[42m\u001b[1m CONFIGURED \u001b[m\u001b[49m \u001b[1mtext\u001b[m (Basic Text Linter)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcsharp\u001b[m (C#)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcpplint\u001b[m (C++ Google's Styleguide)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcppcheck\u001b[m (C++ linter)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mlessc\u001b[m (CSS pre-processor)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcsslint\u001b[m (CSSLint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mchmod\u001b[m (Chmod)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mgjslint\u001b[m (Closure Linter)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcoffeelint\u001b[m (CoffeeLint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mcomposer\u001b[m (Composer Dependency Manager)\n", "\u001b[42m\u001b[1m CONFIGURED \u001b[m\u001b[49m \u001b[1mfilename\u001b[m (Filename)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mgenerated\u001b[m (Generated Code)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mgolint\u001b[m (Golint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mhlint\u001b[m (Haskell Linter)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mjsonlint\u001b[m (JSON Lint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mjson\u001b[m (JSON Lint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mjscs\u001b[m (JavaScript Code Style)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mjshint\u001b[m (JavaScript error checking)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mnolint\u001b[m (Lint Disabler)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mmerge-conflict\u001b[m (Merge Conflicts)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mphpcs\u001b[m (PHP_CodeSniffer)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mphutil-library\u001b[m (Phutil Library Linter)\n", "\u001b[42m\u001b[1m CONFIGURED \u001b[m\u001b[49m \u001b[1mpylint\u001b[m (PyLint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mflake8\u001b[m (Python Flake8 multi-linter)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mpep8\u001b[m (Python PEP 8)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mpyflakes\u001b[m (Python PyFlakes)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mruby\u001b[m (Ruby)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mrubocop\u001b[m (Ruby static code analyzer)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mscript-and-regex\u001b[m (Script and Regex)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mxml\u001b[m (SimpleXML Linter)\n", "\u001b[42m\u001b[1m CONFIGURED \u001b[m\u001b[49m \u001b[1mspelling\u001b[m (Spellchecker)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mxhpast\u001b[m (XHPAST Lint)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mphp\u001b[m (php -l)\n", "\u001b[43m\u001b[1m AVAILABLE \u001b[m\u001b[49m \u001b[1mpuppet-lint\u001b[m (puppet-lint)\n", "(Run `arc linters --verbose` for more details.)\n" ] } ], "source": [ "arc linters" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"linters\": {\n", " \"spelling-linter\": {\n", " \"type\": \"spelling\"\n", " },\n", " \"filename-linter\": {\n", " \"type\": \"filename\"\n", " },\n", " \"text-linter\": {\n", " \"type\": \"text\"\n", " },\n", " \"python-checks\": {\n", " \"type\": \"pylint\",\n", " \"include\": \"(\\\\.py$)\"\n", " }\n", " }\n", "}\n" ] } ], "source": [ "cat .arclint" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Jenkinsfile:4:Auto-Fix: Trailing Whitespace\n", "README:3:Auto-Fix: Trailing Whitespace at EOF\n", "code/mylib.py:1:Advice: Missing Docstring\n", "code/mylib.py:3:Advice: Missing Docstring\n", "code/mylib.py:3:Advice: Old Style Class\n", "code/mylib.py:3:Advice: Too Few Public Methods\n", "code/mylib.py:8:Advice: Invalid Name\n", "code/mylib.py:8:Advice: Missing Docstring\n", "code/mylib.py:8:Advice: No Self Use\n", "code/mylib.py:11:Warning: Mixed Indentation\n", "code/mylib.py:11:Error: Tab Literal\n", "code/mylib.py:11:Warning: Unreachable\n", "code/mylib.py:11:Advice: Invalid Name\n", "code/mylib.py:11:Advice: Missing Docstring\n", "code/mylib.py:11:Warning: Unused Variable\n", "code/mylib.py:11:Warning: Unused Argument\n", "code/mylib.py:12:Warning: Mixed Indentation\n", "code/test.py:1:Advice: Missing Docstring\n", "code/test.py:5:Advice: Missing Docstring\n", "code/test.py:9:Advice: Missing Docstring\n", "code/test.py:12:Advice: Missing Docstring\n", "code/test.py:13:Error: No Member\n" ] }, { "ename": "", "evalue": "2", "output_type": "error", "traceback": [] } ], "source": [ "arc lint --never-apply-patches --output summary --everything" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Arcanist\n", "### Unit-Tests\n", "Using the TAP (Test Anything Protocol) extension of Arcanist, we can report test units result to c4science" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Switched to branch 'feature/arcanist-unittests'\n", "Your branch is up to date with 'origin/feature/arcanist-unittests'.\n", "Submodule path '.arcanist-extensions': checked out 'b89c749a90154676967a1601b62563fed0cbd572'\n", "\u001b[01;34m.\u001b[00m\n", "├── .arcconfig\n", "├── .arclint\n", "├── \u001b[01;34mcode\u001b[00m\n", "│   ├── mylib.py\n", "│   └── \u001b[01;32mtest.py\u001b[00m\n", "├── Dockerfile\n", "├── .gitignore\n", "├── .gitmodules\n", "├── Jenkinsfile\n", "└── README\n", "\n", "1 directory, 9 files\n" ] } ], "source": [ "git checkout feature/arcanist-unittests\n", "git submodule update --init\n", "tree -a -I \".git|.arcanist-extensions\"" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"load\": [\n", " \".arcanist-extensions/tap_test_engine\"\n", " ],\n", "\n", " \"unit.engine\": \"TAPTestEngine\",\n", " \"unit.engine.tap.command\": \"pytest --tap-stream code/test.py\"\n", "}\n" ] } ], "source": [ "cat .arcconfig" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ok 1 - Test.test_capitalize\n", "not ok 2 - Test.test_capitalize_fail\n", "# \n", "# self = \n", "# \n", "# def test_capitalize_fail(self):\n", "# > self.assertTrue(self.m.Capitalize('TeSt') == 'test')\n", "# E AssertionError: False is not true\n", "# \n", "# code/test.py:16: AssertionError\n", "ok 3 - Test.test_return\n", "1..3\n" ] }, { "ename": "", "evalue": "1", "output_type": "error", "traceback": [] } ], "source": [ "pytest --tap-stream ./code/test.py" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ok 1 - Test.test_capitalize\n", "not ok 2 - Test.test_capitalize_fail\n", "# \n", "# self = \n", "# \n", "# def test_capitalize_fail(self):\n", "# > self.assertTrue(self.m.Capitalize('TeSt') == 'test')\n", "# E AssertionError: False is not true\n", "# \n", "# code/test.py:16: AssertionError\n", "ok 3 - Test.test_return\n", "1..3\n", " \u001b[42m\u001b[1m PASS \u001b[m\u001b[49m \u001b[32m <1ms\u001b[39m\u001b[33m★\u001b[39m Test.test_capitalize\n", " \u001b[41m\u001b[1m FAIL \u001b[m\u001b[49m Test.test_capitalize_fail\n", "#\n", " \u001b[42m\u001b[1m PASS \u001b[m\u001b[49m \u001b[32m <1ms\u001b[39m\u001b[33m★\u001b[39m Test.test_return\n" ] }, { "ename": "", "evalue": "2", "output_type": "error", "traceback": [] } ], "source": [ "arc unit --rev HEAD^" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "FROM dkdde/arcanist\n", "RUN apk add --update bash \\\n", " git \\\n", " php7 \\\n", " php7-curl \\\n", " php7-json \\\n", " py2-pip \\\n", " && pip install pytest pytest-tap tap.py \\\n", " && rm -rf /var/cache/apk/*\n", "RUN ln -s /arc/arcanist/bin/arc /bin/arc\n", "\n", "ENTRYPOINT []\n" ] } ], "source": [ "cat Dockerfile" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pipeline {\n", "\n", " agent {\n", " dockerfile true\n", " }\n", "\n", " stages {\n", " stage('run tests') {\n", " steps {\n", " sh 'git submodule update --init'\n", " sh 'arc unit --rev HEAD^'\n", " }\n", " }\n", " }\n", "\n", " post {\n", " always {\n", " archiveArtifacts artifacts: '*.tap'\n", " step([$class: \"TapPublisher\", testResults: \"*.tap\"])\n", " }\n", " }\n", "\n", "}\n" ] } ], "source": [ "cat Jenkinsfile" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![](images/jenkins-tap.png)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# That's all folks !\n", "## Explore, Experiment and Test !\n", "# Questions ?" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" }, "livereveal": { "height": 768, "theme": "serif", "transition": "none", "width": 1024 } }, "nbformat": 4, "nbformat_minor": 2 }