Working with Containers

diff --git a/containers.html b/containers.html index 3f29c2d..70ae1be 100644 --- a/containers.html +++ b/containers.html @@ -1,1374 +1,1372 @@ Working with Containers -

class: center, middle
 
 # Working with Containers
 
 Please open this in your browser to follow along:
 
 `https://goo.gl/hhkKSP`
 
 ---
 
 # Agenda
 
 1. The problem we're solving
 2. Virtual machines vs containers
 3. History of containers
 4. Docker vs Singularity
 5. Singularity workflow
 6. Installing and testing Singularity
 7. Creating and working with containers
 8. Writing a Singularity definition file
 9. Using host resources
 10. Distributing Singularity containers
 11. Docker <-> Singularity interoperability
 12. Extra credits (if time allows)
 
 ---
 
 class: center, middle
 
 # The Problem
 
 ---
 
 # Problem (for developers)
 
 Suppose you're writing some software.  
   It works great on your machine.
 
 However, eventually it has to leave your machine: has to run on your colleague's machine,
   or deployed in its production environment.
 
 It can be a completely different flavour of OS, with a different set of libraries
   and supporting tools.
 
 It can be difficult to test if you accounted for all those variations
   on your own development system.  
   You may have things in your environment you're not even aware of that make a difference.
 
 Your users could also be less technically inclined to deal with dependencies.
   You may wish to decrease this friction.
 
 ---
 
 # Problem (for users)
 
 Suppose you want to run some piece of software.
 
 First off, you really would like some sort of turn-key solution.
   Of course there's none, there's only the source code.
 
 The build instuctions indicate 5-years-old out of date libraries
   on top of a similarly old OS distribution.
 
 And no, the original developer is most certainly no longer available.
 
 You also don't trust this software fully not to mess up your OS.
 
 Or, you want to run it on a remote server for which you don't
   even have the privileges to comfortably install all the dependencies.
 
 ---
 
 # Problem (for researchers)
 
 Suppose you have a piece of scientific software you used
   to obtain some result.
 
 Then someone half across the globe tries to reproduce it,
   and can't get it to run, or worse - is getting different
   results for the same inputs. What is to blame?
 
 Or, even simpler: your group tries to use your software a couple
   of years after you left, and nobody can get it to work.
 
 For a reproducible way to do science with the help of software,
   packaging just the source code might not be enough;
   the environment should also be predictable.
 
 ---
 
 # Problem (for server administrators)
 
 Suppose you have a hundred of users, each requesting certain software.
 
 Some of it needs to be carefully built from scratch,
   as there are no prebuilt packages.
 
 Some of the software works with mutually-incompatible library versions.
   Possibly even known-insecure ones.
 
 Any such software change has to be injected in
   a scheduled maintenance window, but users want it yesterday.
 
 And finally, _you most certainly don't_ trust any of this software
   not to mess up your OS. From experience.
 
 ---
 
 # What would be a solution?
 
 * **A turnkey solution**
 
   A recipe that can build a working instance of your software, reliably and fast.
 
 * **BYOE: Bring Your Own Environment**
 
   A way to capture the prerequisites and environment together with the software.
 
 * **Mitigate security risks**
 
   Provide a measure of isolation between the software running on a system.
   No security is perfect, but some is better than none.
 
 ---
 
 class: center, middle
 
 # The Solution(s)
 
 ---
 
 # Solution: Virtual Machines?
 
 A virtual machine is an isolated instance of a .highlight[whole other "guest" OS] running
   under your "host" OS.
 
 A .highlight[hypervisor] is responsible for handling the situations where this isolation
   causes issues for the guest.
 
 From the point of view of the guest, it runs under its own, dedicated hardware.
   Hence, it's called .highlight[hardware-level virtualization].
 
 Most<sup>*</sup> guest/host OS combinations can run: you can run Windows on Linux,
 Linux on Windows, etc.
 
 ------
 
 \* MacOS being a complicated case due to licensing.
 
 ---
 
 # Virtual Machines: the good parts
 
 * **The BYOE principle is fully realized**
   
   Whatever your environment is, you can package it fully, OS and everything.
 
 * **Security risks are truly minimized**
 
   Very narrow and secured bridge between the guest and the host means little
   opportunity for a bad actor to break out of isolation
 
 * **Easy to precisely measure out resources**
 
   The contained application, together with its OS, has restricted access to
   hardware: you measure out its disk, memory and alotted CPU.
 
 ---
 
 # Virtual Machines: the not so good parts
 
 * **Operational overhead**
 
   For every piece of software, the full underlying OS has to be run,
     and corresponding resources allocated.
 
 * **Setup overhead**
 
   Starting and stopping a virtual machine is not very fast,
   and/or requires saving its state.
 
   Changing the allocated resources can be hard too.
 
 * **Hardware availability**
 
   The isolation between the host and the guest can hinder access to
     specialized hardware on the host system.
 
 ---
 
 # Solution: Containers (on Linux)?
 
 If your host OS is Linux and your software expects Linux, there's
   a more direct and lightweight way to reach similar goals.
 
 Recent kernel advances allow to isolate processes from the rest
   of the system, presenting them with their own view of the system.
 
 You can package entire other Linux distributions, and with the exception
   of the host kernel, all the environment can be different for the process.
 
 From the point of view of the application, it's running on the same hardware
   as the host, hence containers are sometimes called
   .highlight[operating system level virtualization].
 
 ---
 
 # Containers: the good parts
 
 * **Lower operational overhead**
 
   You don't need to run a whole second OS to run an application.
 
 * **Lower startup overhead**
 
   Setup and teardown of a container is much less costly.
 
 * **More hardware flexibility**
 
   You don't have to dedicate a set portion of memory to your VM
     well in advance, or contain your files in a fixed-size filesystem.
 
   Also, the level of isolation is up to you. You may present devices
     on the system directly to containers if you so desire.
 
 ---
 
 # Containers: the not so good parts
 
 * **Kernel compatibility**
 
   Kernel is shared between the host and the container,
     so there may be some incompatibilties.
 
   Plus, container support is (relatively) new, so it needs a recent kernel
     on the host.
 
 * **Security concerns**
 
   The isolation is thinner than in VM case, and kernel of the host OS
     is directly exposed.
 
 * **Linux on Linux**
 
   Containers are inherently a Linux technology. You need a Linux host
     (or a Linux VM) to run containers, and only Linux software can run.
 
 
 ---
 
 # History of containers
 
 The idea of running an application in a different environment is not new to UNIX-like systems.
 
 Perhaps the first effort in that direction is the `chroot` command and concept (1982):
   presenting applications with a different view of the filesystem (a different root directory `/`).
 
 This minimal isolation was improved in in FreeBSD with `jail` (2000),
   separating other resources (processes, users)
   and restricting how applications can interact with each other and the kernel.
 
 Linux developed facilities for isolating and controlling access to some processes
   with namespaces (2002) and cgroups (2007).
 
 Those facilities led to creation of solutions for containerization, notably LXC (2008),
   Docker (2013) and Singularity (2016).
 
 ---
 
 # Docker
 
 Docker came about in 2013 and since has been on a meteoric rise
   as the golden standard for containerization technology.
 
 A huge amount of tools is built around Docker to build, run,
   orchestrate and integrate Docker containers.
 
 Many cloud service providers can directly integrate Docker containers.
   Docker claims x26 resource efficiency improvement at cloud scale.
 
 Docker encourages splitting software into microservice chunks
   that can be portably used as needed.
 
 ---
 
 # Docker concerns
 
 Docker uses a pretty complicated model of images/volumes/metadata,
   orchestrating swarms of those containers to work together,
   and it not always very transparent with how those are stored.
 
 Also, isolation features require superuser privileges;
   Docker has a persistent daemon running with those privileges
   and many container operations require root as well.
 
 Both of those issues make Docker undesirable in applications
   where you don't wholly own the computing resource - HPC environments.
 
 Out of those concerns, and out of scientific community, came Singularity.
 
 ---
 
 # Singularity
 
 Singularity is quite similar in principles to Docker. In fact,
   it's pretty straightforward to convert a Docker container
   to a Singularity image.
 
 Singularity uses a monolithic, image-file based approach.
   Instead of dynamically overlaid "layers" of Docker, you have
   a single file you can build once and simply copy over to the target system.
 
 ---
 
 # Singularity and root privileges
 
 The privilege problem was a concern from the ground-up, and solved by having a
   `setuid`-enabled binary that can accomplish container startup - and drop
   privileges completely as soon as practical.
 
 Privilege elevation inside a container is impossible: `setuid` mechanism
   is disabled inside the container, so to be root inside,
   you have to be root outside. And users don't need explicit root
   access to operate containers (at least after the initial build).
 
 ---
 
 # Singularity and HPC
 
 Thanks to the above improvements over Docker, HPC cluster operators
   are much more welcoming to the idea of Singularity support.
 
 As a result of a joint Pipeline Interoperability project between Swiss Science IT groups,
   we have a [set of guidelines](https://goo.gl/hHHYLL)
   for Singularity deployment and use by scientific community.
 
 Also as part of this effort, the UniBE Linux cluser UBELIX started support Singularity.
 
 Once your software is packaged in Singularity, it should work across
   all Science IT platforms supporting the technology.
 
 ---
 
 # Singularity workflow
 
 .minusmargin[
   ![:scale 100%](assets/singularity-2.4-flow.png)
 ]
 
 1. Interactively develop steps to construct a container
 2. Describe the steps in a recipe.
 2. Build an immutable container on own machine.
 3. Deploy this container in the production environment.
 
 ---
 
 class: center, middle
 
 # Working with Singularity
 
 ---
 
 # Installing Singularity
 
 Installing Singularity from source is probably preferred,
   as it's still a relatively new piece of software.
 
 Instructions at: https://www.sylabs.io/guides/2.6/user-guide/installation.html#install-a-specific-release
 (7 lines of shell commands, use `VER=2.6.0`)
 
 On Ubuntu, packages required are: `build-essential squashfs-tools libarchive-dev`.
 
 .exercise[
   If you want to try installing Singularity on your Linux system, follow the build instructions.
   
   You will need root access!
 
   If you're using the remote training machine, skip this step.
 ]
 
 ---
 
 # Using Singularity
 
 If you followed build instructions, you should now have `singularity`
   available from the shell.
 
 ```
 user@host:~$ singularity --version
 2.6.0-dist
 ```
 
 The general format of Singularity commands is:
 
 ```
 singularity [<global flags>] <command> [<command flags>] [<arguments>]
 ```
 
 Singularity is pretty sensitive to the order of those.
 
 Use `singularity help [<command>]` to check built-in help.
 
 You can find the configuration of Singularity under `/etc/singularity`
 if you used the default prefixes.
 
 ---
 
 # Container images
 
 A Singularity image is, for practical purposes, a filesystem tree
   that will be presented to the applications running inside it.
 
 A Docker container is built with a series of _layers_ that are
   stacked upon each other to form the filesystem. Singularity
   collapses those into a single, portable file.
 
 A container needs to be somehow bootstrapped to contain a base
   operating system before further modifications can be made.
 
 ---
 
 # Pulling Docker images
 
 The simplest way of obtaining a working Singularity image
   is to pull it from either Docker Hub or Singularity Hub.
 
 Let's try it with CentOS 6:
 
 ```
 user@host:~$ singularity pull docker://centos:6
 ```
 
 This will download the layers of the Docker container to your
 machine and assemble them into an image.
 
 ---
 
 # Pulling Docker images
 
 ```
 user@host:~$ singularity pull docker://centos:6
 WARNING: pull for Docker Hub is not guaranteed to produce the
 WARNING: same image on repeated pull. Use Singularity Registry
 WARNING: (shub://) to pull exactly equivalent images.
 Docker image path: index.docker.io/library/centos:6
 Cache folder set to /home/ubuntu/.singularity/docker
 Importing: base Singularity environment
 Exploding layer: sha256:1c8f9aa56c90ab24207ff5ca6b853bdbffb40b7801055d0b9c934cb16b5b43dd.tar.gz
 Exploding layer: sha256:b98c5d595dc1be386927f4f6e75dc3801f043e38902dcdd7235abf0c79b404b3.tar.gz
 WARNING: Building container as an unprivileged user. If you run this container as root
 WARNING: it may be missing some functionality.
 Building Singularity image...
 Singularity container built: ./centos-6.simg
 Cleaning up...
 ```
 
 Note that this .highlight[does not require `sudo` or Docker]!
 
 .exercise[
   Pull the CentOS 6 image from Dockerhub with the above command
 ]
 
 ---
 
 # Entering shell in the container
 
 To test our freshly-created container, we can invoke an interactive shell
   to explore it with .highlight[`shell`]:
 
 ```
 user@host:~$ singularity shell centos-6.simg
 Singularity: Invoking an interactive shell within container...
 
 Singularity centos-6.simg:~>
 ```
 
 At this point, you're within the environment of the container.
 
 We can verify we're "running" CentOS:
 
 ```
 Singularity centos-6.simg:~> cat /etc/centos-release
 CentOS release 6.9 (Final)
 ```
 
 ---
 
 # User/group within the container
 
 Inside the container, we are the same user:
 
 ```
 Singularity centos-6.simg:~> whoami
 user
 Singularity centos-6.simg:~> exit
 user@host:~$ whoami
 user
 ```
 
 We will also have the same groups. That way, if any host resources are mounted
   in the container, we'll have the same access privileges.
 
 ---
 
 # Root within the container
 
 If we launched `singularity` with `sudo`, we would be `root` inside the container.
 
 ```
 user@host:~$ sudo singularity shell centos-6.simg
 Singularity: Invoking an interactive shell within container...
 
 Singularity centos-6.simg:~> whoami
 root
 ```
 
 Note that the `pull` command complained that building without root does not guarantee
   that `root` inside will work as expected.
 
 **Most importantly:** `setuid` mechanism will not work within the container.
   Once launched as non-root, no command can elevate your privileges.
 
 ---
 
 # Default mounts
 
 Additionally, by default:
 
 * our home folder,
 * `/tmp`,
 * `/dev`,
 * the folder we've invoked Singularity from
 
 are accessible inside the container:
 ```
 user@host:~$ singularity shell centos-6.simg
 Singularity centos-6.simg:~> ls ~
 [..lists home folder..]
 Singularity centos-6.simg:~> touch ~/test_container
 Singularity centos-6.simg:~> exit
 user@host:~$ ls ~/test_container
 /home/user/test_container
 ```
 
 The current working directory inside the container is the same
   as outside at launch time.
 
 ---
 
 # Running a command directly
 
 Besides the interactive shell, we can execute any command
   inside the container directly with .highlight[`exec`]:
 
 ```
 user@host:~$ singularity exec centos-6.simg cat /etc/centos-release
 CentOS release 6.9 (Final)
 ```
 
 .exercise[
   Invoke the `python` interpreter with `exec`.
 
   Compare the version with the host system.
 ]
 
 ---
 
 # STDIO with container processes
 
 Standard input/output are processed as normal by Singularity.
 You can redirect them:
 
 ```
 ubuntu@host:~$ singularity exec centos-6.simg echo Boo > ~/test_container
 ubuntu@host:~$ singularity exec centos-6.simg cat < ~/test_container
 Boo
 ```
 
 You can use containers in pipelines:
 
 ```
 $ singularity exec centos-6.simg echo Boo | singularity exec centos-6.simg cat
 Boo
 ```
 
 .exercise[
   Count the number of words in host's `ls /etc`'s output using container's copy of `wc`,
   then the other way around. Hint:
 
   ```
   ls /etc | wc -w
   ```
 ]
 
 ---
 
 # Modifying the container
 
 Let's try to install some software in the container. We will need root:
 
 ```
 user@host:~$ sudo singularity shell centos-6.simg
 Singularity centos-6.simg:~> whoami
 root
 Singularity centos-6.simg:~> fortune
 bash: fortune: command not found
 ```
 
 We cannot appeal to the wisdom of `fortune`, and that will not do! Let's install it.
 
 ```
 Singularity centos-6.simg:~> yum -y --enablerepo=extras install epel-release
 [...]
 Singularity centos-6.simg:~> yum -y install fortune-mod
 [...]
 Singularity centos-6.simg:~> fortune
 [a random fortune cookie text]
 ```
 
 ---
 
 # Modifying the container
 
 Perfect! Let's exit the container.
 
 ```
 Singularity centos-6.simg:~> exit
 exit
 user@host:~$
 ```
 
 We should now be able to use fortune at will:
 
 ```
 user@host:~$ singularity exec centos-6.simg fortune
 ```
 
 .exercise[
   Try the above!
 ]
 
 ---
 
 # Modifying the container?..
 
 Perfect! Let's exit the container.
 
 ```
 Singularity centos-6.simg:~> exit
 exit
 user@host:~$
 ```
 
 We should now be able to use fortune at will:
 
 ```
 user@host:~$ singularity exec centos-6.simg fortune
 /.singularity.d/actions/exec: line 9: exec: fortune: not found
 ```
 
 Wait, what?
 
 ```
 user@host:~$ singularity shell centos-6.simg
 Singularity centos-6.simg:~> fortune
 bash: fortune: command not found
 ```
 
 What happened to our modifications?
 
 ---
 
 # Images and overlays
 
 The image Singularity creates is normally read-only.
 
 While any changes that happen to bind-mounted folders from the host will persist,
   any changes to the container's filesystem itself are not saved.
 
 Before 2.4, Singularity would just produce write errors; now, it .highlight[temporarily]
   applies the changes (as an "overlay"), which are then discarded when your session ends.
 
 ---
 
 # Writing to the image
 
 `shell` and `exec` have a `--writable` flag.
 
 Looks like what we need! Let's try again:
 
 ```
 user@host:~$ sudo singularity shell --writable centos-6.simg
 ```
 
 ---
 
 # Writing to the image?..
 
 `shell` and `exec` have a `--writable` flag.
 
 Looks like what we need! Let's try again:
 
 ```
 user@host:~$ sudo singularity shell --writable centos-6.simg
 ERROR  : Unable to open squashfs image in read-write mode: Read-only file syst
 em
 ABORT  : Retval = 255
 ```
 
 Well, that didn't work either.
 
 Why? And how _do_ we write to an image?
 
 ---
 
 # Mutable vs immutable containers
 
 Since 2.4, Singularity has 2 types of containers:
 
 1. Mutable containers. Those are either loose collections of files in a directory,
   or ext3-formatted images (deprecated).
 
 2. Immutable containers. Those are compressed, read-only squashfs images.
 
 The purpose of mutable containers is to interactively develop a "recipe" for a container,
   that will be then converted to an immutable "final" form.
 
 `pull` will create immutable containers; for mutable ones, we need `build`.
 
 ---
 
 # Creating a writable container
 
 Let's remove our immutable container and build a sandbox container:
 
 ```
 $ rm centos-6.simg
 $ sudo singularity build --sandbox centos-writable docker://centos:6
 ```
 
 This is just a collection of files in `./centos-writable` folder
   that represents the root of the filesystem:
 
 ```
 user@host:~$ ls centos-writable/
 bin  dev  environment  etc  home  lib  lib64  lost+found  media  mnt  opt
 proc  root  sbin  selinux  singularity  srv  sys  tmp  usr  var
 ```
 
 ---
 
 # Writing to a container, finally:
 
 We should now be able to enter it **in writable mode** and install software
   (fortune, and text editors for the next step):
 
 ```
 user@host:~$ sudo singularity shell --writable centos-writable
 Singularity centos-writable:~> yum -y --enablerepo=extras install epel-release
 [...]
 Singularity centos-writable:~> yum -y install fortune-mod vim nano
 [...]
 Singularity centos-writable:~> exit
 user@host:~$ singularity exec centos-writable fortune
 [some long-awaited wisdom of a fortune cookie]
 ```
 
 ---
 
 # Giving container purpose
 
 A container can have a "default" command which is run without specifying it.
 
 Inside the container, it's `/singularity`. Let's try modifying it:
 
 ```
 user@host:~$ sudo singularity exec -w centos-writable vim /singularity
 ```
 
 By default you'll see the following:
 
 ```bash
 #!/bin/sh
 
 exec /bin/bash "$@"
 ```
 
 This is a script that will pass all arguments to `/bin/bash`.
 
 ---
 
 # Giving container purpose
 
 We installed `fortune`, so let's use that instead:
 
 ```bash
 #!/bin/sh
 
 exec /usr/bin/fortune "$@"
 ```
 
 .exercise[
   Make the same modification to your container.
 ]
 
 Now we can invoke it with .highlight[`run`]:
 
 ```
 user@host:~$ singularity run centos-writable
 [..some wisdom or humor..]
 ```
 
 ---
 
 # Converting to final container
 
 One way to produce a "final" container is to convert it from the sandbox version:
 
 ```
 user@host:~$ sudo singularity build fortune.simg centos-writable
 [...]
 ```
 
 Now we can test our container:
 
 ```
 user@host:~$ singularity run fortune.simg
 [..some more wisdom..]
 ```
 
 ---
 
 # Running a container directly
 
 Note that the container file is executable:
 
 ```
 user@host:~$ ls -lh fortune.simg
 -rwxr-xr-x 1 root root 99M Feb 30 13:37 fortune.simg
 ```
 
 If we run it directly, it's the same as invoking `run`:
 
 ```
 user@host:~$ ./fortune.simg
 [..a cracking joke..]
 ```
 
 This does require to have `singularity` installed on the host, however, and is just a convenience.
 
 ---
 
 # Making the container reproducible
 
 Instead of taking some base image and making changes to it by hand,
   we want to make this build process reproducible.
 
 This is achieved with definition files called .highlight[Recipes].
 
 Let's try to retrace out steps to obtain a fortune-telling CentOS.
 
 .exercise[
   Open a file called `fortune.def` in an editor, and prepare to copy along.
 ]
 
 ---
 
 # Bootstrapping
 
 The definition file starts with a header section.
 
 The key part of it is the `Bootstrap:` configuration,
   which defines how we obtain the "base" image.
 
 There are 3 currently types of bootstrap methods:
 
 * using `yum`/`apt`/`pacman` etc. on the host system to bootstrap a similar one
 * pull an image: `docker` (from Docker Hub) or `shub` (from Singularity Hub)
 * `localimage` to base off another image on your computer
 
 We'll be using the Docker method.
 
 ```
 Bootstrap: docker
 From: centos:6
 ```
 
 ---
 
 # Setting up the container
 
 There are 2 sections for setup commands (essentially shell scripts):
 
 1. .highlight[`%setup`] for commands to be executed .highlight[outside the container].
 
   You can use `$SINGULARITY_ROOTFS` to access the container's filesystem,
     as it is mounted on the host during the build.
 
 2. .highlight[`%post`] for commands to be executed .highlight[inside] the container.
 
   This is a good place to set up the OS, such as installing packages.
 
 ---
 
 # Setting up the container
 
 Let's save the name of the build host and install `fortune`:
 
 ```
 Bootstrap: docker
 From: centos:6
 
 %setup
   hostname -f > $SINGULARITY_ROOTFS/etc/build_host
 
 %post
   yum -y --enablerepo=extras install epel-release
   yum -y install fortune-mod
 ```
 
 ---
 
 # Adding files to the container
 
 An additional section, .highlight[`%files`], allows to copy files or folders to the container.
 
 **We won't be using it here**, but the format is very similar to  `cp`,
 with sources being outside and the final destination being inside the container:
 
 ```
 %files
   some/file /some/other/file some/path/
   some/directory some/path/
 ```
 
 Note that this happens _after_ `%post`. If you need the files earlier, copy them manually in `%setup`.
 
 ---
 
 # Setting up the environment
 
 You can specify a script to be sourced when something is run in the container.
 
 This goes to the .highlight[`%environment`] section. Treat it like `.bash_profile`.
 
 ```
 %environment
   export HELLO=World
 ```
 
 Note that by defaut, the host environment variables are passed to the container.
 
 To disable it, use `-e` when running the container.
 
 ---
 
 # Setting up the runscript
 
 The runscript (`/singularity`) is specified in the `%runscript` section.
 
 Let's use the file we copied at `%setup` and run `fortune`:
 
 ```
 %runscript
   read host < /etc/build_host
   echo "Hello, $HELLO! Fortune Teller, built by $host"
   exec /usr/bin/fortune "$@"
 ```
 
 ---
 
 # Testing the built image
 
 You can specify commands to be run at the end of the build process inside the container
   to perform sanity checks.
 
 Use `%test` section for this:
 
 ```
 %test
   test -f /etc/build_host
   test -x /usr/bin/fortune
 ```
 
 All commands must return successfully or the build will fail.
 
 ---
 
 # The whole definition file
 
 ```
 Bootstrap: docker
 From: centos:6
 
 %setup
   hostname -f > $SINGULARITY_ROOTFS/etc/build_host
 %post
   yum -y --enablerepo=extras install epel-release
   yum -y install fortune-mod
 %environment
   export HELLO="World"
 %runscript
   read host < /etc/build_host
   echo "Hello, $HELLO! Fortune Teller, built by $host"
   exec /usr/bin/fortune "$@"
 %test
   test -f /etc/build_host
   test -x /usr/bin/fortune
 ```
 
 .exercise[
   Check that your `fortune.def` is the same as above.
 ]
 
 ---
 
 # Building a container from definition
 
 To fill a container using a definition file, we invoke `build`:
 
 ```
 user@host:~$ rm fortune.simg
 user@host:~$ sudo singularity build fortune.simg fortune.def
 [...]
 ```
 
 .exercise[
 1. Bootstrap the image as shown above.
 
 2. Test running it directly.
 ]
 
 ---
 
 # Inspecting a container
 
 If a container was built from a Recipe, it has some metadata you can read:
 
 ```
 user@host:~$ singularity inspect fortune.simg
 {
     "org.label-schema.usage.singularity.deffile.bootstrap": "docker",
     "vendor": "CentOS",
     "name": "CentOS Base Image",
 [...]
 ```
 
 You can inspect the original Recipe:
 
 ```
 user@host:~$ singularity inspect -d fortune.simg
 Bootstrap: docker
 From: centos:6
 %setup
   hostname -f > $SINGULARITY_ROOTFS/etc/build_host
 [...]
 ```
 
 See `singularity help inspect` for more options, and `/.singularity.d/`
   inside the container to see how it's all stored.
 
 ---
 
 # Host resources
 
 A container can have more host resources exposed.
 
 For providing access to more directories,
   one can specify bind options at runtime with `-B`:
 
 ```
 $ singularity run -B source[:destination[:mode]] container.simg
 ```
 
 where .highlight[source] is the path on the host,
   .highlight[destination] is the path in a container (if different)
   and .highlight[mode] is optionally `ro` if you don't want to give write access.
 
 Of course, more than one bind can be specified.  
 Note that you can't specify this configuration in the container!
 
 System administrators may specify binds that apply to all containers (e.g. `/scratch`).
 
 ---
 
 # Host resources
 
 Additionally, devices on the host can be exposed, e.g. the GPU; but you need to make
   sure that the guest has the appropriate drivers. One solution is to bind the drivers on the container.
 
 For Nvidia CUDA applications specifically, Singularity supports the `--nv` flag, which looks for
   specific libraries on the host and binds them in the container.
 
 -----
 
 OpenMPI should also work, provided the libraries on the host and in the container are sufficiently close.
 
 If set up correctly, it should work normally with `mpirun`:
 
 ```
 $ mpirun -np 20 singularity run mpi_job.simg
 ```
 
 ---
 
 # Fuller isolation
 
 By default, a container is allowed a lot of "windows" into the host system (dictated by Singularity configuration).
 
 For an untrusted container, you can further restrict this with options like `--contain`, `--containall`.
 
 In this case, you have to manually define where standard binds like the home folder or `/tmp` point.
 
 See `singularity help run` for more information.
 
 ---
 
 exclude: true
 
 # TODO: Lightweight OSes for containers
 
 ---
 
 # Distributing the container
 
 Using the container after creation on another Linux machine is simple:
   you simply copy the image file there.
 
 Note that you can't just run the image file on a host without Singularity installed!
 
 .exercise[
   Test the above, by trying to run `fortune.simg` inside itself.
 ]
 
 You can easily integrate Singularity with the usual scheduler scripts (e.g. Slurm).
 
 ---
 
 # Using Singularity Hub
 
 Singularity Hub allows you to cloud-build your containers from Bootstrap files,
   which you can then simply `pull` on a target host.
 
 This requires a GitHub repository with a `Singularity` definition file.
 
 After creating an account and connecting to the GitHub account,
   you can select a repository and branches to be built.
 
 Afterwards, you can pull the result:
 
 ```
 user@host:~$ singularity pull shub://kav2k/fortune
 [...]
 Done. Container is at: /home/user/kav2k-fortune-master.simg
 user@host:~$ ./kav2k-fortune-master.simg
 Hello, World! Fortune Teller, built by shub-builder-1450-kav2k-fortune-[...]
 ```
 
 ---
 
 # Running on UBELIX
 
 From a practical standpoint, we want to use the container technology on UBELIX.
 
 Let's try with our toy container:
 
 ```
 user@host:~$ ssh username@submit.unibe.ch
 username@submit01:~$ singularity pull shub://kav2k/fortune
 username@submit01:~$ sbatch -J fortune-test -t 00:00:10
 --mem-per-cpu 100M --cpus 1 --wrap "./kav2k-fortune-master-latest.simg"
 ```
 
 ---
 
 # Docker and Singularity
 
 Instead of writing a Singularity file, you may write a `Dockerfile`,
   build a Docker container and convert that.
 
 Pros:
 
 * More portable: for some, using Docker or some other container solution is preferable.
 * Easier private hosting: private version of Singularity Hub, `sregistry`, is still not mature.
 
 Cons:
 
 * Blackbox: Singularity understands less about the build process, in terms of container metadata.
 * Source of bugs: Sadly, Docker pull is not perfect and may result in bugs.
 * Complexity: Extra tool to learn if you don't know Docker.
 
 ---
 
 # Docker -> Singularity
 
 If you have a Docker image you want to convert to Singularity,
   you have at least two options:
 
 1. Upload the image to a Docker Registry (such as Docker Hub)
   and `pull`/`Bootstrap` from there.
 
 2. Use a local copy of a Docker registry to not rely on external services
 
 ---
 
 class: center, middle
 
 # "Extra credit" topics
 
 ---
 
 # Reducing container size
 
 Using traditional Linux distributions, even in minimal configurations,
   can still be an overkill for running a single application.
 
 One can reduce container size by clearing various artifacts of the build
   process, such as package manager caches.
 
 Alternatively, one can use minimal Linux distributions, such as Alpine Linux,
   as a base for containers, though compatibility needs extra testing.
 
 ```
 $ ll -h
 -rwxr-xr-x  1 user group  66M Jun 25 15:04 centos-6.simg*
 -rwxr-xr-x  1 user group 2.0M Jun 25 16:08 alpine.simg*
 ```
 
 ---
 
 # Singularity Instances
 
 Running daemon-like persistent services with Singularity (such as a web server)
   can conveniently be done with the concept of Instances.
 
 A `%startscript` section of the recipe describes what service to launch,
   which subsequently works with `instance.*` commands:
 
 ```
 $ singularity instance.start nginx.simg web
 $ singularity instance.list
 INSTANCE NAME    PID      CONTAINER IMAGE
 web              790      /home/mibauer/nginx.simg
 $ singularity instance.stop web
 ```
 
 While an instance is running, the standard commands like `shell` and `exec`
   work with an `instance://` namespace.
 
 ---
 
 # Persistent Overlays
 
 As was shown before, once you create a "final" image it can no longer be changed,
   and all writes go to a temporary overlay.
 
 If needed, such changes can be persisted in a separate permanent overlay file:
 
 ```
 $ singularity image.create my-overlay.img
 $ sudo singularity shell --overlay my-overlay.img ubuntu.simg
 [ changes made will persist if launched with the same --overlay ]
 ```
 
 ---
 
 # SCI-F
 
 One of the approaches for building scientific pipelines is bundling several tools
   in a single "toolset" container.
 
 SCI-F is a proposed standard for discovering and managing tools within
   such modular containers.
 
 Definition file can have several sections, e.g.:
 
 ```
 %appenv foo
     BEST_GUY=foo
     export BEST_GUY
 
 %appenv bar
     BEST_GUY=bar
     export BEST_GUY
 
 %apprun foo
     echo The best guy is $BEST_GUY
 
 %apprun bar
     echo The best guy is $BEST_GUY
 ```
 
 ---
 
 # SCI-F
 
 You can then discover the apps bundled and run them:
 
 ```
 $ singularity apps foobar.simg
 bar
 foo
 $ singularity run --app bar foobar.simg
 The best guy is bar
 ```
 
 More sections can be made app-specific, including providing a `help` description:
 
 ```
 $ singularity help --app fortune moo.simg
 fortune is the best app
 ```
 
 ---
 
 # Singularity Checks
 
 A container check is a utility script that can verify a container.
 
 Example uses:
 
 * Making sure no leftover artifacts from the build process remains (e.g. root's bash history)
 
 * Testing for common vulnerabilities
 
 * Custom checks for your specific environment
 
 ```
 $ singularity check --tag clean ubuntu.img
 ```
 
 ---
 
 class: center, middle
 
 # Questions?
 
 ---
 
 exclude: true
 
 # Runner script
 
 We have developed a small runner script that allows to specify
   a lot of "external" metadata (like bind commands or image size)
   in a single configuration file alongside other bootstrap files,
   as well as scripts to run on the host to run/validate the container.
 
 You can install it with `pip install singularity-pipeline`
 
 Documentation and samples available at https://c4science.ch/diffusion/2915/browse/master/UniBe/
 
 Consider it still in alpha, but it helps writing down the metadata
   for a scientific pipeline.

- \ No newline at end of file diff --git a/css/linux_1.css b/css/linux_1.css deleted file mode 100644 index e69de29..0000000 diff --git a/js/linux_1.js b/js/linux_1.js deleted file mode 100644 index e69de29..0000000 diff --git a/linux_1.html b/linux_1.html index ba82479..fe2bd04 100644 --- a/linux_1.html +++ b/linux_1.html @@ -1,2203 +1,2201 @@ Introduction to Linux, Part 1 -

class: center, middle
 
 # Introduction to Linux
 
 ## Part I
 
 `https://goo.gl/8t6byZ`
 
 ---
 
 # Agenda
 
 1. What is Linux?
 2. Linux interface: GUI vs CLI
 3. Connecting to a remote Linux system
 4. Linux directory structure
 5. Moving and looking around
 6. Reading and writing files
 7. Organizing files and folders
 8. Moving data from/to a remote Linux system
 
 ---
 
 class: center, middle
 
 # What is Linux?
 
 ![](assets/Tux.png)
 
 ---
 
 class: center
 
 # What is Linux?
 
 --
 
 <br>
 
 The most common answer you'll hear is:
 
 ## "Linux is an operating system"
 
 --
 
 <br>
 
 But what does this mean?
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td>System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td class="bluebg">System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td rowspan="2"><strong>Operating system</strong><br>Windows, Linux, MacOS, Android, ...</td>
     </tr>
     <tr>
       <td class="bluebg">Kernel</td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td class="bluebg">System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td rowspan="2"><strong>Linux</strong></td>
     </tr>
     <tr>
       <td class="bluebg">Kernel</td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ## In practice, we call this part "Linux"
 
 ---
 
 # Linux? Wait, I also heard "UNIX"?
 
 UNIX is the name of an operating system from 1970 that pioneered concepts that
   will form the basis of Linux (and other OSes) today.
 
 More importantly, it introduced a set of conventions that its descendents follow.
   A system that follows them is called "UNIX-like".
 
 Most of what you learn here will easily transfer to other UNIX-like OSes (e.g. macOS).
 
 ---
 
 ![:scale 100%](assets/Unix_timeline.en.svg)
 
 ---
 
 class: center, two-columns
 
 # User Interface
 
 GUI                 | CLI
 ------------------- | -------------
 Graphical Interface | Command Line
 ![:scale 80%](assets/ubuntu-desktop.png) | ![:scale 100%](assets/terminal.png)
                     | Some synonyms:
                     | "Shell", "Terminal", "TTY"
 
 ---
 
 # Command Line Interface
 
 ```
 user@host:~ $ cowsay "Command Line Interface"
  ________________________
 < Command Line Interface >
  ------------------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||
 ```
 
 --
 
 * Input, output, and commands are **text**.
 
 --
 
 * Easy on the computer: can run on any hardware.
 
 --
 
 * Network-friendly: a few bytes of text vs realtime stream of images / GUI updates => tool of choice for remote access.
 
 --
 
 * Scripting/automation-friendly: text is easier to manipulate.
 
 --
 
 * Expert-friendly, but _beginner-unfriendly._
 
 ---
 
 # Connecting to a remote Linux system
 
 The standard tool to connect to a remote system is `ssh`.
 
 .acronym[
 SSH: .highlight[S]ecure .highlight[Sh]ell
 ]
 
 It securely connects you to a remote system.  
 Communication is encrypted, both parties are authenticated.
 
 First, you will need to log in to the system.
 
 If your credentials are accepted, it creates a new shell for you.
 
 It is then displayed on your screen and controlled by your keyboard,
 relayed over the network.
 
 ---
 
 # Connecting from MacOS / Linux:
 
 Good news: you already have a terminal and `ssh` of your own!
 
 --
 
 First, open the terminal:
 
 * For MacOS, it's accessible from Launchpad, Utilities.
 * For Linux GUI, usually look for a program called Terminal.
 
 --
 
 Then, you need to input the command to connect to a remote host:
 
 ```
 local.user@local:~ $ ssh user@remote
 
 [..some mutual* authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 # Connecting from Windows:
 
 You will need an SSH client. Standard one: PuTTY
 
 .center[.middle[
 ![:scale 20%](assets/putty-logo.jpg) ![:scale 50%](assets/putty-main.jpg)
 ]]
 
 Download the appropriate installer: https://goo.gl/pHFReU
 
 ---
 
 # Connecting from Windows:
 
 * Make sure "Connection type: SSH" is selected.
 * Put the remote's host name / IP in the form.
 * Select "Open"
 
 A terminal window will open..
 
 ```
 [..some mutual* authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 background-image: url(assets/tofu.svg)
 background-position: top right
 background-size: 40%
 
 # Mutual authentication?
 
 (and what's up with this side picture?)
 
 ---
 
 background-image: url(assets/tofu.svg)
 background-position: top right
 background-size: 40%
 
 # Mutual authentication?
 
 SSH authenticates both parties:
 
 * Client to server
   * Username + password
   * Username + cryptographic key
   * Something else!
 * Server to client
   * The server has a cryptographic key to prove its identity
 
 --
 
 The first time you connect, you need to explicitly say you trust the (previously unknown) server.
 
 On subsequent connections, SSH will verify that you are still connecting to a server with the same key,
 and will warn you before login credentials are transmitted if you aren't.
 
 --
 
 This is called TOFU (.highlight[T]rust .highlight[O]n .highlight[F]irst .highlight[U]se).
 
 ---
 
 layout: true
 
 # Mutual authentication
 
 So, the first time you connect to a new server, you should _expect_ a warning you need to confirm:
 
 ---
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)?
 ```
 
 ---
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)? yes
 ```
 
 ---
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)? yes
 Warning: Permanently added 'remote,11.22.33.44' (ECDSA) to the list of known 
 hosts.
 
 [..some client authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 In Windows/PuTTY:
 
 .center[
   ![:scale 60%](assets/putty-server-auth.png)
 ]
 
 ---
 
 layout: false
 
 # Hands-on time: Connect to a server
 
 Using your Campus account username/password, use SSH/PuTTY to connect to UBELIX at `submit.unibe.ch`.
 
 ```
 ssh user@submit.unibe.ch
 ```
 
 A reminder, PuTTY can be obtained from https://goo.gl/pHFReU
 
 ---
 
 # Greetings from a shell
 
 ![:scale 100%](assets/greetings_shell.jpg)
 
 ---
 
 # Greetings from a shell
 
 After connecting, you will be greeted with something like this:
 
 ```
 user@remote:~ $
 ```
 
 --
 
 What you see is the interface of the .highlight[shell]: a text-based interface that allows you to launch other programs with commands.
 
 --
 
 .mnemonic[
 It's called a shell .highlight[prompt] since it's .highlight[prompting] you to enter a command.
 ]
 
 The prompt contains a short summary of current state of the shell.
 
 ---
 
 # Anatomy of a prompt
 
 The prompt looks like this:
 
 ```
 user@remote:~ $
 ```
 
 This may vary slightly from system to system, and is fully configurable, but this is the typical form.
 
 --
 
 This form of the prompt answers 3 questions:
 
 * Who are you? .highlight[Username `user`]
 
 * Where are you? .highlight[Hostname `remote`]
 
 * Where in the filesystem are you? .highlight[`~`] (explained later)
 
 --
 
 Terminating the prompt is (traditionally) a .highlight[`$`] character:
 it delimits where your input goes.
 
 ---
 
 layout: true
 
 # Taking command
 
 The shell expects a textual command; most of the time you type the command and press [ENTER] to commit it.
 
 Let's try this (in slow motion)!
 
 ---
 
 ```
 user@remote:~ $
 
 ⠀
 ```
 
 ---
 
 ```
 user@remote:~ $ whoami
 
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits
 
 ---
 
 ```
 user@remote:~ $ whoami
 
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits
 2. Pressing [ENTER]. The shell will process the command (launch the program `whoami`)
 
 ---
 
 ```
 user@remote:~ $ whoami
 user
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits.
 2. Pressing [ENTER]. The shell will process the command — launch the program `whoami`.
 3. The program will take over input/output — in this case, it will output your username).
 
 ---
 
 ```
 user@remote:~ $ whoami
 user
 user@remote:~ $ ⠀
 ```
 
 1. Typing in "whoami" as the shell waits.
 2. Pressing [ENTER]. The shell will process the command — launch the program `whoami`.
 3. The program will take over input/output — in this case, it will output your username).
 4. The program terminates, and control returns to the shell; it shows a new prompt.
 
 ---
 
 layout: false
 
 # Try it!
 
 Here's a few commands for you to try:
 
 ```
 whoami
 
 echo Hello!
 
 pwd
 
 ls -l
 
 date
 
 sleep 3
 
 clear
 
 history 5
 ```
 
 Each should do something and return you to the shell prompt.  
 Can you guess what they do?
 
 Note that you can use up/down arrows to access/repeat previous commands.
 
 ---
 
 # Safety first, or emergency exits!
 
 So far every command we encountered automatically returned control back to the shell.
 
 But what if a program is stuck, or expecting some input and you're not sure what to do?
 
 Typical shortcuts to stop / quit a program:
 
 * `Ctrl` + `C` (also called .highlight[interrupt])
 * `Esc` (from ".highlight[escape]")
 * `q` (from ".highlight[q]uit")
 * `Ctrl` + `D` (.highlight[end of input], in case a program is waiting)
 
 If you try those, usually you'll either exit the program or get some hint on how to do it.
 
 `Ctrl` is sometimes denoted as `^`, e.g. `^C` for `Ctrl`+`C`.
 
 ---
 
 exclude: true
 
 # Anatomy of a shell command
 
 The shell expects input. What does it (typically) look like?
 
 ```
 user@remote:~ $ program -f --option abc 123
 ```
 
 Here, `program` is the .highlight[command] being executed, and the rest is the list of its .highlight[arguments]:
 
 1. `-f`
 2. `--option`
 3. `abc`
 4. `123`
 
 Arguments that start with `-` or `--` are often called .highlight[flags] or .highlight[switches] and traditionally change
 some options of the command.
 
 ---
 
 # The Working Directory
 
 One of the commands you executed, `pwd`, printed a directory path (of your home directory, by default):
 
 ```
 akashev@submit01:~ $ pwd
 /home/ubelix/math/akashev
 ```
 
 --
 
 .mnemonic[
   `pwd` stands for .highlight[P]rint .highlight[W]orking .highlight[D]irectory
 ]
 
 --
 
 Whenever you use the shell, there is a concept of the current (or "working") directory.
 This affects how commands search for files and how they interpret paths.
 
 Think of it as of "where" you are: if a server is a building you're in, a working directory
   is the room you're in within that building.
 
 
 ---
 
 # The Working Directory
 
 One of the commands you executed, `pwd`, printed a directory path (of your home directory, by default):
 
 ```
 akashev@submit01:~ $ pwd
 /home/ubelix/math/akashev
 ```
 
 Usually, this information is printed in the shell prompt itself, to remind you of the current state.
 
 --
 
 In this example it's `~`, which represents the .highlight[home directory].
 
 --
 
 Here's how it would look if you were somewhere else, for example in `/var/log`:
 
 ```
 akashev@submit01:/var/log $
 ```
 
 ---
 
 # UNIX directory structure
 
 If you're reading this tutorial, you likely already know that files are normally organized
   into nested "directories" (or "folders"). For example, on Windows you may have such a path:
 
 ```
 C:\folder\subfolder\file
 ```
 
 On Linux, paths looks similarly:
 
 ```
 /home/user/folder/subfolder/file
 ```
 
 ---
 
 # UNIX directory structure
 
 ```
 /folder/subfolder/file
 ```
 
 * a file .highlight[`file`]
 * inside a directory .highlight[`subfolder`]
 * which is inside a directory .highlight[`folder`]
 * which itself is inside the .highlight[root directory] `/`
 
 ![](assets/file-tree.svg)
 
 ---
 
 # The `/` as directory separator
 
 Forward slashes (`/`) separate the folders in the path.  
 Using multiple is valid, so the following is the same file:
 
 ```
 /home/user/folder/subfolder/file
 ///home/user///folder/subfolder//file
 ```
 
 --
 
 A path to a regular file never ends in `/`, e.g. this is not valid:
 
 ```
 /home/user/folder/subfolder/file/
 ```
 
 --
 
 Directories can be referred to with or without the final `/`:
 
 ```
 /home/user/folder/subfolder
 /home/user/folder/subfolder/
 ```
 
 --
 
 Root directory is special: `/` is its only name.
 
 ---
 
 # Absolute and relative paths
 
 If a path starts with `/`, it's an .highlight[absolute] path that starts at root:
 
 ```
 /home/user/folder/subfolder/file
 ```
 
 --
 
 If it does not, then it's a .highlight[relative] path
 that starts at the current working directory instead of `/`.
 
 If the current working directory is
 
 ```
 /home/user/folder
 ```
 
 then the following paths point to the same file:
 
 ```
 /home/user/folder/subfolder/file
 subfolder/file
 ```
 
 ---
 
 # Absolute and relative paths
 
 ```
 /home/user/folder/subfolder/file
 subfolder/file
 ```
 
 ![:scale 60%](assets/File-tree-2.svg)
 
 ---
 
 # Special folders `.` and `..`
 
 There are 2 special folders inside each folder: `.` and `..`
 
 * `.` points to the folder itself.
 
 ```
 /home/user/folder/subfolder/./file
 ```
 
 --
 
 * `..` points to one folder "up" in the path. At root, it points to root itself.
 
 ```
 /home/user/another_folder/../folder/file
 /home/../../home/user/folder/file
 ```
 --
 
 It's mostly important for relative paths:
 
 ```
 # From /home/user/another_folder
 ../folder/file
 ```
 
 ---
 
 # Special folders `.` and `..`
 
 From `/home/user/another_folder`
 
 ```
 /home/user/folder/subfolder/file
 .././folder/subfolder/file
 ```
 
 ![:scale 60%](assets/File-tree-3.svg)
 
 ---
 
 # Home directories
 
 Each user has a .highlight[home directory] assigned.
 
 It acts as your default working directory.
 
 By convention, its path usually starts with `/home/`
 and ends with your username:
 
 ```
 /home/<maybe something else>/username
 ```
 
 It's frequently referred to as `~`:
 
 ```
 /home/username/folder/file
 ~/folder/file
 ```
 
 You can even refer to others' home folder with `~username`:
 
 ```
 /home/someone/file
 ~someone/file
 ```
 
 ---
 
 # Quiz time! [1/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /scratch/folder/B
 Target:             /scratch/folder/A/a
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>~/../scratch/folder/A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~userA/../../scratch/folder/A/a</code></td>
     </tr>
     <tr>
       <td class="N"><code>A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>../A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/scratch/./folder/A/a</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Quiz time! [2/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /home/userA/temp
 Target:             ../../userB/folder/file
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>/home/userB/userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/home/userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="N"><code>~/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~/../userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~userB/folder/file</code></td>
     </tr>
   </table>
 </div>
 
 ---
 
 # Quiz time! [3/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /home/userA/folder
 Target:             /home/userA/folder/file
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>./file</code></td>
     </tr>
     <tr>
       <td class="N"><code>~/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/home/userA/folder/subfolder/../file</code></td>
     </tr>
   </table>
 </div>
 
 ---
 
 # Preparing for training
 
 Please execute the following command to add the exercises to your home folder:
 
 ```
 $ wget https://scits.math.unibe.ch/script -O - | /bin/bash
 ```
 
 (That's a capital O and spaces are significant)
 
 This should be the only time you don't understand what you're doing;
   and by the end of Part II you should understand it.
 
 Hint: On a Swiss German keyboard, `|` is <kbd>AltGr</kbd> + <kbd>7</kbd>
 
 ---
 
 # Moving around
 
 Now that we know:
 
 * Files and directories are organised in a tree
 * There's a "current"/working directory that we are in
 
 we need to learn to move around in that tree.
 
 --
 
 For that, we need the `cd` command:
 
 ```
 user@remote:~ $ cd scits-training
 user@remote:~/scits-training $ pwd
 /home/username/scits-training
 user@remote:~/scits-training $
 ```
 
 .acronym[
 `cd` stands for ".highlight[C]hange .highlight[D]irectory"
 ]
 
 ---
 
 # Moving around 
 
 The general format of the command is `cd DESTINATION`, where `DESTINATION` is a path
 (relative or absolute) to a directory.
 
 ```
 user@remote:~ $ cd scits-training
 user@remote:~/scits-training $ cd /usr/local/bin
 user@remote:/usr/local/bin $
 ```
 
 --
 
 To go "back up", one uses the special `..` directory:
 
 ```
 user@remote:/var/local/bin $ cd ..
 user@remote:/usr/local $ cd ../..
 user@remote:/ $
 ```
 
 --
 
 To go to your home directory, you can use `~`:
 
 ```
 user@remote:/ $ cd ~
 user@remote:~ $
 ```
 
 ---
 
 # `cd` shortcuts
 
 There are two useful tricks when using `cd`:
 
 "`cd -`" goes back to the previous directory you were in:
 
 ```
 user@remote:~ $ cd -
 user@remote:/ $
 ```
 
 And "`cd`" without arguments goes to your home folder:
 ```
 user@remote:/ $ cd
 user@remote:~ $
 ```
 
 ---
 
 # Tab-completion
 
 This is a good point to introduce a helpful CLI tool: .highlight[tab completion]
 
 When entering a command, you can press the `[Tab]` key to suggest a command, or path,
 based on already entered input.
 
 ```
 user@remote:~ $ cd scits-training/a
 ```
 
 Pressing `[Tab]` now completes the name, since it's the only one that matches the beginning:
 
 ```
 user@remote:~ $ cd scits-training/animals/
 ```
 
 (continues on next slide)
 
 ---
 
 layout: true
 
 # Tab-completion
 
 ```
 user@remote:~ $ cd scits-training/animals/
 ```
 
 Pressing `[Tab]` once again won't change anything, since there are mutiple choices for completion;
 however, if it is pressed again, it shows possibilities:
 
 ```
 user@remote:~ $ cd scits-training/animals/
 Aardvark/ Badger/
 user@remote:~ $ cd scits-training/animals/
 ```
 
 ---
 
 ---
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/A
 ```
 
 ---
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/Aardvark/
 ```
 
 ---
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/Aardvark/
 user@remote:~/scits-training/animals/Aardvark/ $ 
 ```
 
 ---
 
 layout: false
 
 # Looking around
 
 To look around in a UNIX filesystem, you use the `ls` command:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls
 description  empty_file  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 .mnemonic[
 `ls` stands for .highlight[list]
 ]
 
 This lists the names for contents of the working directory.
 
 --
 
 We can specify another folder to look at:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls ../Badger/
 Arctonyx  Meles  Mellivora  Melogale  Mydaus
 ```
 
 ---
 
 # Looking around (in depth)
 
 To show more information, we can use the `-l` (for .highlight[l]ong) flag:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -l
 total 25640
 -rw-r--r-- 1 username groupname 26214400 Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname      754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname        0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname     4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 Important information from this output:
 
 * `-rw-r--r--` is called the .highlight[mode] (explained in Part II).
   * `d` denotes .highlight[directory] in this example.
   * `rw-r--r--` deals with permissions for the files.
 * `username` and `groupname` are .highlight[owners] of the file.
 * The number after `groupname` is the .highlight[size] (in bytes) of the file.
   * Important: for folders, it's not the size of all contents.
 * The date/time after the size is the .highlight[modification date].
 
 ---
 
 # Looking around (as puny humans)
 
 One can use the flag `-h` (for .highlight[h]uman-readable) for more familiar size units:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -l -h
 total 26M
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 Single-letter flags in commands can often be combined:  
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -lh
 total 26M
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 ---
 
 # Looking around (into hidden corners)
 
 Another often-used flag is `-a` (for .highlight[a]ll): it lists contents with names that start with a dot `.`
 which are normally hidden in UNIX.
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -a
 <span style="color:#3030ff;font-weight:bold">.</span>  <span style="color:#3030ff;font-weight:bold">..</span>  big_file  description  empty_file  .hidden  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 As usual, it can be combined with others:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -lah
 total 26M
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">.</span>
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">..</span>
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 .hidden
 ```
 
 ---
 
 
 # Looking around (in orderly fashion)
 
 By default, files are ordered by name.
 
 This behavior can be changed with flags; here are some examples:
 
 * `-r` .highlight[r]everses the sort order.
 * `-S` sorts files by .highlight[s]ize.
 * `-t` sorts files by modification .highlight[t]ime.
 * `-X` sorts files by filename e.highlight[x]tension, e.g. `png` in `image.png`.
 
 As usual, this can be combined with the previous ones.
 
 .exercise[
 List files in `Aardvark` by increasing size.
 ]
 
 ---
 
 # I'm never going to remember this!
 
 .center[
 ![](assets/tar.png)
 ]
 
 ## Good news: you don't have to.
 
 As long as you remember the command's name, you can look up its correct usage from the terminal itself.
 
 .small[Image credit: https://xkcd.com/1168/]
 
 ---
 
 layout: true
 
 # Getting help
 
 Some common methods of getting help:
 
 ---
 
 * Many programs support .highlight[`--help` flag] to print out their usage instructions:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls --help
 Usage: ls [OPTION]... [FILE]...
 List information about the FILEs (the current directory by default).
 Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
 
 Mandatory arguments to long options are mandatory for short options too.
   -a, --all                  do not ignore entries starting with .
 [...]
 ```
 
 ---
 
 * For most programs, you can look up their .highlight[manual file] with `man`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ man ls
 ```
 
 Instead of just outputting the text and returning, you'll enter a mode for showing long files.
 
 Look around using arrow keys and `PgDn`/`PgUp`.
 
 Remember the hints on how to exit (here, it's `q`).
 
 You can search a `man` page for "something" with `/something` and just `/` to go to the next find.
 
 ---
 
 * Some commands are not separate programs, but are .highlight[built into the shell], e.g. `cd`.
   For those, you can use `help`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ help cd
 ```
 
 --
 
 You can see what `help` can help with as well:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ help
 ```
 
 ---
 
 layout: false
 
 # Try out `man`
 
 Try opening the manual for `ls`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ man ls
 ```
 
 Reminders:
 
 * You can search a `man` page for "something" with `/something` and `n` to go to the next find.
 
 * To exit, you can use `q`.
 
 .exercise[
 Try searching for the meaning of `-R` flag, and try to use it.
 ]
 
 ---
 
 # Reading files
 
 We know how to look around the filesystem (with `ls`) and how to move around (with `cd`).
 
 However, we still need to access the contents of files.
 
 There are many ways to do that, I'll show a few more common ones.
 
 ---
 
 background-image: url(assets/cat.svg)
 background-position: top 5% right 30%
 background-size: 15%
 
 # Simple file reading
 
 The simplest program to read the file is `cat`
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls
 big_file  description  empty_file  naming  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 user@remote:~/scits-training/animals/Aardvark/ $ cat description
 The aardvark (ARD-vark; Orycteropus afer) is a medium-sized, burrowing,
 [...]
 ```
 
 .mnemonic[
 `cat` comes from the word "con.highlight[cat]enate",  
 which means joining things together in a series.
 ]
 
 .exercise[
 What happens if we call `cat` with two filenames?
 
 ```cat description naming```
 ]
 
 ---
 
 background-image: url(assets/cat_long.svg)
 background-position: top 5% right 20%
 background-size: 25%
 
 # File is too long!
 
 Sometimes a file is too long to be comfortably read with `cat`
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ cd ../../numbers/
 user@remote:~/scits-training/numbers/ $ cat hundred
 1
 2
 [...]
 99
 100
 ```
 
 A hundred lines is too much to fit into the terminal window.
 
 While you can scroll to look through the output, sometimes files are much longer than that.
 
 We can display only parts of the file, or use a program that allows to navigate a file.
 
 ---
 
 background-image: url(assets/cat_parts.svg)
 background-position: top 5% right 30%
 background-size: 15%
 
 # Parts of a cat?
 
 If a cat is too long, perhaps we only need to look at its beginning (`head`)
 or end (`tail`):
 
 ```
 user@remote:~/scits-training/numbers/ $ head hundred
 1
 [...]
 10
 ```
 
 ```
 user@remote:~/scits-training/numbers/ $ tail hundred
 91
 [...]
 100
 ```
 
 Those commands display the first and last 10 lines of a file, respectively.
 
 .mnemonic[
   Remembering `cat` together with `head` and `tail` may help.
 ]
 
 ---
 
 # Self-help test
 
 Of course, you can look up other options with the self-help methods like `man`.
 
 .exercise[
 Use one of the help methods (`man head` or `head --help`) to learn how to display 5 lines instead of 10 with `head`.
 ]
 
 Hint: it will be a flag that should go before the filename.
 
 --
 
 **Answer:** `-n 5`, `-n5` or `--lines=5`
 
 ```
 user@remote:~/scits-training/numbers/ $ head -n 5 hundred
 1
 2
 3
 4
 5
 ```
 
 ---
 
 # The file is too long, show less
 
 One way to navigate a big file is `less`:
 
 ```
 user@remote:~/scits-training/numbers/ $ less hundred
 ```
 
 You will recognize this interface, since `man` also uses `less`.
 
 Commands to try:
 * .highlight[Arrow keys] to scroll line by line
 * .highlight[`PgUp`] / .highlight[`PgDn`] to scroll screen by screen
 * .highlight[`/something`] to search for "something"
 * .highlight[`n`] to go to next found "something", .highlight[`N`] to go back
 * .highlight[`>`] to go to the end of the file, .highlight[`<`] to go to the beginning
 * .highlight[`h`] to show help
 * .highlight[`q`] to quit
 
 ---
 
 # Modifying files
 
 Besides reading, we need to be able to create and modify files.
 
 There are many editors available, and which one is "best" can lead to [hot debate](https://xkcd.com/378/).
 
 We will mention and briefly explain two editors that are likely to be installed on any system you encounter nowadays.
 
 * `nano`
 * `vim`
 
 ---
 
 # `nano`
 
 ```
 user@remote:~/scits-training/numbers/ $ nano hundred
 
 <span style="background:white;color:black;">GNU nano 2.5.3               File: hundred                                    </span>
 
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
                                <span style="background:white;color:black;">[ Read 100 lines ]</span>
 <span style="background:white;color:black;">^G</span> Get Help  <span style="background:white;color:black;">^O</span> Write Out <span style="background:white;color:black;">^W</span> Where Is  <span style="background:white;color:black;">^K</span> Cut Text  <span style="background:white;color:black;">^J</span> Justify   <span style="background:white;color:black;">^C</span> Cur Pos
 <span style="background:white;color:black;">^X</span> Exit      <span style="background:white;color:black;">^R</span> Read File <span style="background:white;color:black;">^\</span> Replace   <span style="background:white;color:black;">^U</span> Uncut Text<span style="background:white;color:black;">^T</span> To Spell  <span style="background:white;color:black;">^_</span> Go To Line
 ```
 
 ---
 
 # `nano`
 
 `nano` is a small and simple editor which helpfully shows its commands at the bottom (reminder, `^` means `Ctrl`):
 
 ```
 <span style="background:white;color:black;">^G</span> Get Help  <span style="background:white;color:black;">^O</span> Write Out <span style="background:white;color:black;">^W</span> Where Is  <span style="background:white;color:black;">^K</span> Cut Text  <span style="background:white;color:black;">^J</span> Justify   <span style="background:white;color:black;">^C</span> Cur Pos
 <span style="background:white;color:black;">^X</span> Exit      <span style="background:white;color:black;">^R</span> Read File <span style="background:white;color:black;">^\</span> Replace   <span style="background:white;color:black;">^U</span> Uncut Text<span style="background:white;color:black;">^T</span> To Spell  <span style="background:white;color:black;">^_</span> Go To Line
 ```
 
 You can use arrow keys to move around, input text as normal from where the cursor is.
 
 Key commands:
 
 * `Ctrl` + `W` ".highlight[w]here is" for searching the file
 * `Ctrl` + `O` "write .highlight[o]ut" to save changes
 * `Ctrl` + `X` "e.highlight[x]it" to get back to the shell
 
 ---
 
 # Try `nano`
 
 .exercise[
 1. Open a new file, `ten`, with `nano`:
 
   ```
   user@remote:~/scits-training/numbers/ $ nano ten
   ```
 
 2. Add numbers from 1 to 10 to it, on separate lines
 
 3. Save and exit `nano`
 
 4. Verify what's in the file using `cat`
 ]
 
 ---
 
 # `vim`
 
 `vim` (or, technically, "Vi IMproved") is one of two "Swiss knife" editors that
 most Linux professionals prefer to use (the other one being `emacs`).
 
 `vim` is available almost everywhere, and with proper configuration can do
 very sophisticated things.
 
 With power comes complexity, but for basic editing one doesn't have to remember a lot.
 
 If you wish to (later) explore `vim`, you can go through its built-in tutorial:
 
 ```
 vimtutor
 ```
 
 ---
 
 .center[
   [![:scale 60%](assets/exiting_vim.png)](http://www.commitstrip.com/en/2017/05/29/trapped/)
 ]
 
 ---
 
 # Organizing files and folders
 
 To recap, you should now be able to:
 
 * Navigate the file tree (with `cd`)
 * List folder contents (with `ls`)
 * Read and write files (with `nano`)
 
 Our goal now is:
 
 * Make new folders
 * To move and copy files and folders around
 * Delete files and folders
 
 ---
 
 # Creating new folders
 
 To create new folders, use the `mkdir` command:
 
 ```
 user@remote:~/scits-training/numbers/ $ cd ..
 user@remote:~/scits-training/ $ ls
 <span style="color:#3030ff;font-weight:bold">animals</span>  <span style="color:#3030ff;font-weight:bold">numbers</span>
 user@remote:~/scits-training/ $ mkdir new-folder
 user@remote:~/scits-training/ $ ls
 <span style="color:#3030ff;font-weight:bold">animals</span>  <span style="color:#3030ff;font-weight:bold">new-folder</span>  <span style="color:#3030ff;font-weight:bold">numbers</span>
 ```
 
 .mnemonic[
   .highlight[`mkdir`] stands for .highlight[make dir]ectory
 ]
 
 .exercise[
 1. Create `new-folder` as shown above
 
 2. Create directory `subfolder` inside it
 
 3. Verify with `ls`
 ]
 
 ---
 
 # Creating new folders
 
 `mkdir` will fail if the folder already exists:
 
 ```
 user@remote:~/scits-training/ $ mkdir new-folder
 mkdir: cannot create directory ‘new-folder’: File exists
 ```
 
 Using it with `-p` means "create if needed", and also works with chains of directories:
 
 ```
 user@remote:~/scits-training/ $ mkdir -p new-folder/subfolder/subsubfolder
 user@remote:~/scits-training/ $ ls -R new-folder
 new-folder:
 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 
 new-folder/subfolder:
 <span style="color:#3030ff;font-weight:bold">subsubfolder</span>
 
 new-folder/subfolder/subsubfolder:
 user@remote:~/scits-training/ $
 ```
 
 ---
 
 # Moving files
 
 Move operations can be broken down into two cases:
 
 1. Moving files and folders between folders:
 
   `folder1/something` → `folder2/something`
 
 2. Renaming files and folders: 
 
   `something` → `other`
 
   Technically, it's "moving" from old name to new.
 
 Both cases are served with the `mv` command.
 
 .mnemonic[
   .highlight[`mv`] stands for .highlight[move]
 ]
 
 ---
 
 # Preparing for exercises
 
 ```
 user@remote:~/scits-training/ $ cd moving
 user@remote:~/scits-training/moving $ ls
 <span style="color:#3030ff;font-weight:bold">source</span>  <span style="color:#3030ff;font-weight:bold">destination</span>
 user@remote:~/scits-training/moving $ ls source
 A1  A10  A11  A12  A2  A3  A4  A5  A6  A7  A8  A9  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 user@remote:~/scits-training/moving $ ls source/subfolder
 B1  B2  B3  B4  B5  B6  B7  B8  B9
 user@remote:~/scits-training/moving $ ls destination
 user@remote:~/scits-training/moving $
 ```
 
 ---
 
 # Moving files
 
 To move something to another folder: .highlight[`mv NAME DESTINATION`],
 as long as the .highlight[`DESTINATION` is a directory that exists].
 
 ```
 user@remote:~/scits-training/moving $ mv source/A2 destination
 ```
 
 You can specify multiple things to move at the same time, including folders:
 
 ```
 $ mv source/A3 source/subfolder destination
 ```
 
 Moves both `source/A3` and `source/subfolder` into `destination`.
 
 .exercise[
   Move `subfolder` back into `source`
 ]
 
 ---
 
 # Renaming
 
 Renaming is easy: `mv OLDNAME NEWNAME`, if `NEWNAME` is *not* a directory.
 
 For example, let's rename `destination` to `dest`:
 
 ```
 user@remote:~/scits-training/moving $ mv destination dest
 ```
 
 If you're renaming something in another folder, you must specify the path twice:
 
 ```
 $ mv source/A4 source/A40
 ```
 
 .exercise[
 1. Rename `dest` back into `destination`
 
 2. Rename `source/subfolder/B1` into `source/subfolder/B10`
 ]
 
 ---
 
 # Move + rename
 
 .exercise[
 Try the following:
 
 ```
 user@remote:~/scits-training/moving $ mv source/A5 A50
 ```
 
 Use `ls` to understand what happened (`-R` may help)
 ]
 
 --
 
 **Answer:** Since there is no path for the second name, it moved into the current directory
 and got renamed:
 
 `~/scits-training/moving/source/A5`
 
 ↓
 
 `~/scits-training/moving/A50`
 
 ---
 
 # Copying
 
 Copying is done with `cp`
 
 .mnemonic[
   .highlight[`cp`] stands for .highlight[copy].
 ]
 
 Syntax is the same:
 
 * For copying to another directory, `cp NAME DESTINATION`
 * For copying to another name, `cp OLDNAME NEWNAME`
 
 .exercise[
 1. Copy `source/A6` and `source/A7` into `destination`
 
 2. Copy `source/A6` into `source/A66`
 ]
 
 ---
 
 # Copying folders
 
 `cp`, unlike `mv`, will not copy directories by default:
 
 ```
 $ cp source/subfolder destination
 cp: omitting directory 'source/subfolder'
 ```
 
 You need to use `-R` to copy folders together with their content
 
 ```
 $ cp -R source/subfolder destination
 ```
 
 .mnemonic[
 .highlight[`-R`] stands for .highlight[r]ecursive
 ]
 
 ---
 
 # Deleting
 
 To remove files or folders, use `rm`
 
 .mnemonic[
   .highlight[`rm`] stands for .highlight[rem]ove
 ]
 
 * `rm NAME` to remove a file
 * `rm -r FOLDER` to remove a folder
 
 You can pass several names at once:
 
 ```
 $ rm destination/subfolder/B1 destination/subfolder/B2
 ```
 
 ---
 
 # `rm` is unrecoverable!
 
 When you delete files and folders with `rm`, you should be aware that there is no concept of "Trash".
 
 Anything you delete (or overwrite) is lost with no easy way to recover.
 
 You can use a flag `-i` to ask before any destructive operation.
 
 ```
 user@remote:~/scits-training/moving $ cp -i -R source/subfolder destination
 cp: overwrite 'destination/subfolder/B1'?
 ```
 
 On the other hand, sometimes you want to override those confirmations, especially for `rm` – you can do it with `-f`.
 
 .mnemonic[
   .highlight[`-i`] stands for .highlight[interactive]  
   .highlight[`-f`] stands for .highlight[force]
 ]
 
 ---
 
 # Wildcards
 
 There are many A-files in `source`:
 
 ```
 user@remote:~/scits-training/moving $ ls source
 A1  A10  A11  A12  A40  A6  A66  A7  A8  A9  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 We may want to copy them all at once. We can use wildcards:
 
 * .highlight[`*`] in a name means "any amount of any characters"
   * For example, .highlight[`A*`] can mean `A`, `A1` and `A10`
 * .highlight[`?`] in a name means "any single character"
   * For example, .highlight[`A?`] can mean `A1`, `A6` but not `A10`
 
 The wildcards will .highlight[not jump through directories]:
 
 * .highlight[`*1`] can mean `A1`, `A11`, but not `subfolder/B1`
 * .highlight[`\*/\*`] can match `subfolder/B1`
 
 ---
 
 # Wildcard quiz (1/3)
 
 Which of the following names match the pattern .highlight[`A\*a\*`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>AAa</code></td>
     </tr>
     <tr>
       <td class="N"><code>A/a</code></td>
     </tr>
     <tr>
       <td class="N"><code>aaA</code></td>
     </tr>
     <tr>
       <td class="N"><code>CBAcba</code></td>
     </tr>
     <tr>
       <td class="N"><code>abcABC</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Wildcard quiz (2/3)
 
 Which of the following names match the pattern .highlight[`A?a?`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>AAa</code></td>
     </tr>
     <tr>
       <td class="Y"><code>AAaa</code></td>
     </tr>
     <tr>
       <td class="Y"><code>Aaaa</code></td>
     </tr>
     <tr>
       <td class="N"><code>AAaaa</code></td>
     </tr>
     <tr>
       <td class="N"><code>aAAaa</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Wildcard quiz (3/3)
 
 Which of the following patterns match the name .highlight[`A110`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>A*</code></td>
     </tr>
     <tr>
       <td class="Y"><code>*</code></td>
     </tr>
     <tr>
       <td class="Y"><code>*A*</code></td>
     </tr>
     <tr>
       <td class="N"><code>*A</code></td>
     </tr>
     <tr>
       <td class="Y"><code>A???</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Using wildcards
 
 Putting a name with a wildcard is equivalent to putting several names:
 
 ```
 $ cp source/A6* destination
 ```
 
 is equivalent to
 
 ```
 $ cp source/A6 source/A66 destination
 ```
 
 So, you can use wildcards in any command that expects multiple files.
 
 ---
 
 # Try wildcards
 
 Use wildcards to do the following, from `~/scits-training/moving`:
 
 .exercise[
 1. List all files starting with `A` inside `source/` (use `ls` with a pattern).
 
 2. Copy all files starting with `B` from `source/subfolder` into `destination`.
 
 3. Move all files starting with `A1` from `source` into `destination`.
 
 4. Delete all files starting with `A` from `destination`.
 ]
 
 ---
 
 # Moving data in and out
 
 So far we have moved the data around on the system itself.
 
 It doesn't help if you want to load external data or
 download the results of your programs.
 
 --
 
 Perhaps, it's your own system and you have access to cloud storage
 or external storage devices.
 
 Sometimes, you have a shared network folder between your computer
 and the target system.
 
 But how to do it, if your only interface to the server is SSH?
 
 --
 
 We will cover two ways:
 
 1. Downloading data from the Internet with `wget`
 
 2. Copying data between computers with `scp`
 
 ---
 
 # Downloading from the shell
 
 Sometimes, the data you need is a file on the Internet.
 
 `wget` is the simplest-to-use tool for it:
 
 ```
 user@remote:~/scits-training/moving $ cd ..
 user@remote:~/scits-training/ $ wget https://example.com/
 --2017-09-12 12:00:00--  https://example.com/
 Resolving example.com (example.com)... 93.184.216.34
 Connecting to example.com (example.com)|93.184.216.34|:443... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 1270 (1.2K) [text/html]
 Saving to: ‘index.html’
 
 100%[======================================>] 1,270       --.-K/s   in 0s
 
 2017-09-12 12:00:00 (36.2 MB/s) - ‘index.html’ saved [1270/1270]
 
 user@remote:~/scits-training/ $ tail index.html
 ```
 
 .mnemonic[
   .highlight[`wget`] stands for .highlight[Web get]
 ]
 
 ---
 
 # Downloading from the shell
 
 Notable option: renaming the file immediately.
 
 * .highlight[-O] (for .highlight[o]utput) chooses a specific file to write to
 
 ```
 $ wget https://tools.ietf.org/rfc/rfc1149.txt -O april.txt
 [...]
 $ less april.txt
 ```
 
 As usual, use `man wget` to see more options.
 
 It can work with HTTP/HTTPS/FTP-hosted files.
 
 ---
 
 # Transferring files between systems
 
 To send files between two computers using SSH, the simplest command is `scp`.
 
 .mnemonic[
   .highlight[`scp`] stands for .highlight[secure copy]
 ]
 
 `scp` behaves a lot like `cp`, but you can provide locations on other computers.
 
 How to use `scp` on your own machine depends on the OS.
 
 ---
 
 # `scp` on Windows
 
 While PuTTY includes a command-line client `pscp` with the same functions,
 it may be better to use a GUI client `WinSCP`.
 
 It can be downloaded from https://winscp.net/
 
 You can then connect using SCP (or SFTP) with your normal credentials
 and transfer files between your PC and the remote:
 
 ![](assets/winscp.png)
 
 ---
 
 # `scp` on Linux / MacOS
 
 From your .highlight[local] terminal, you can transfer a file from UBELIX:
 
 ```
 user@local:~ $ scp user@submit.unibe.ch:~/scits-training/numbers/hundred .
 [..some authentication..]
 user@local:~ $ less hundred
 ```
 
 `scp`'s parameters work similarly to `cp`, but you can refer to files on other systems
 by adding `user@remote:` to the path.
 
 It works both ways, and can rename as well:
 ```
 $ scp hundred user@submit.unibe.ch:~/scits-training/numbers/another_hundred
 ```
 
 ---
 
 # Moving data in and out
 
 .exercise[
 1. Copy all of the `B`-files from `moving/source/subfolder` to your computer with one command (use wildcards).
 
 2. Copy some folder from your computer to the home folder of the remote system (use `-r`).
 ]
 
 --
 
 In addition to `scp`, there's a command that works better
 for repeatedly copying large folders with small changes: .highlight[`rsync`].
 
 It will not be covered here, but look up information on it if it's your use case.

- \ No newline at end of file diff --git a/linux_2.html b/linux_2.html index 644c70b..f47ec50 100644 --- a/linux_2.html +++ b/linux_2.html @@ -1,1407 +1,1405 @@ Introduction to Linux, Part II -

class: center, middle
 
 # Introduction to Linux
 
 ## Part II
 
 `https://goo.gl/cypvZ1`
 
 ---
 
 # Agenda
 
 1. Linux resources you can use
 2. Standard input/output and its redirection
 3. UNIX pipelines
 4. Background processes 
 5. Durable sessions with `screen`
 6. File ownership and permissions
 7. Shell scripting basics
 8. Environment and customization
 9. (Extra credits)
 
 ---
 
 # What Linux resources can I use?
 
 To do development and run light workloads:
 
 * Your own computer may already run Linux.
 
 * You can install Linux in a virtual machine.
 
   I recommend Virtualbox for personal use.
 
 * If you're running Windows 10, you can install  
   [Windows Subsystem for Linux](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux)
 
 ---
 
 # What if it's not enough?
 
 To create persistent services:
 
 * Ask your group's sysadmin for servers/VM resources.
 
 * UniBe Informatikdienste offers virtual machines.
 
 * Cloud resources: [SWITCHengines](https://www.switch.ch/engines/), other cloud services.
 
 To run heavy calculations:
 
 * [UBELIX Linux cluster](https://docs.id.unibe.ch/ubelix/ubelix-101).
 
 * Your group may have in-house infrastructure.
 
 * Again, cloud services.
 
 ---
 
 # Prepare for the tutorial
 
 You should be either running Linux, or connected to a Linux system.
 
 Log in to UBELIX, or inform me that you need access to another system.
 
 Execute the following to set up training (if you haven't already):
 ```
 $ wget https://scits.math.unibe.ch/script -O - | /bin/bash
 ```
 
 ---
 
 # Processes and their input/output
 
 When you're connected to a Linux system,
   what you normally see is the shell prompt, awaiting input:
 
 ```
 user@host:~ $
 ```
 
 A .highlight[process] called shell is responsible for input/output at this moment.
 
 By default, the input is either a keyboard connected to the system,
   or your keypresses being relayed over the network.
 
 The output, by default, is the screen connected to the system,
   or text being relayed for display over the network.
 
 ---
 
 # Processes and their input/output
 
 Those are called .highlight[standard input] and .highlight[standard output],
   or .highlight[STDIN/STDOUT].
 
 .center[
   ![](assets/shell_stdio.svg)
 ]
 
 --
 
 If the user issues a command that calls another program, the shell
   creates a .highlight[child process] and attaches the input/output to it.
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 The shell will wait until the program terminates,
   after which STDIN/STDOUT get reattached and a prompt is displayed.
 
 ---
 
 # Redirecting output
 
 Sometimes, we want to capture what a command is outputting to the screen.
 
 For example, suppose that we want to save into a file the directory structure returned by `ls -R`:
 
 ```
 user@host:~ $ ls -R ~/scits-training
 scits-training:
 animals  moving  numbers  io  scripts
 [...]
 user@host:~ $
 ```
 
 We can instruct the shell to .highlight[redirect] the standard output:
 
 .center[
   ![](assets/stdout_redirect.svg)
 ]
 
 ---
 
 # Redirecting output
 
 To redirect the output, we add .highlight[`> FILE`] to the command:
 
 ```
 user@host:~ $ cd ~/scits-training/io
 user@host:~/scits-training/io $ ls -R ~/scits-training > listing
 user@host:~/scits-training/io $ cat listing
 scits-training:
 animals  moving  numbers  io  scripts
 [...]
 ```
 
 This will overwrite the contents of `FILE` (if any) with the output of the command.
 
 The file will be created, if it does not exist yet.
 
 ---
 
 # Appending output
 
 Sometimes we don't want to overwrite (sometimes called "clobber") the file
   with new contents and add them to the end instead.
 
 To do that, use .highlight[`>> FILE`] instead of `> FILE`.
 
 .exercise[
 1. Try running the command `date` to see what it outputs.
 
 2. Run `date` 3 times, appending the output to a file `date.log`.
 
 3. Verify with `cat` that the file contains 3 records.
 ]
 
 ---
 
 # Error output
 
 Try saving the output of a command with errors
 and you'll see that it still outputs to the screen:
 
 ```
 user@host:~/scits-training/io $ ls , > listing
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $
 ```
 
 --
 
 This is intentional: Linux actually has two output streams for
   its command line, STDOUT for normal data and .highlight[STDERR] for errors.
 
 .center[
   ![](assets/stderr.svg)
 ]
 
 This simplifies debugging: errors are separate from data.
 
 ---
 
 # Error output
 
 STDERR is not redirected when using `>` or `>>`:
 
 .center[
   ![](assets/stderr_noredirect.svg)
 ]
 
 --
 
 It's possible to redirect it as well with .highlight[`2>`] or .highlight[`2>>`]:
 
 ```
 user@host:~/scits-training/io $ ls .. , > listing 2> errors
 user@host:~/scits-training/io $ cat errors
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $ cat listing
 ..:
 animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Discarding output
 
 Sometimes we don't need output at all.
 
 In this case, we can redirect it to a special file, .highlight[`/dev/null`]
 
 It's a .highlight[device] that will accept any input, discarding it immediately.
 
 --
 
 For example, one might want to silence errors:
 
 ```
 user@host:~/scits-training/io $ ls .. , 2> /dev/null
 ..:
 animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Interactive input
 
 Most commands we've seen don't require any interactive input.
 
 `tr` (for .highlight[translate]) is a command that transforms its input:
   it substitutes some characters with others.
 
 For example, `tr 'a-z' 'A-Z'` would translate all lowercase letters into uppercase.
 
 --
 
 Let's try this:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 ```
 
 ---
 
 # Interactive input
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 But enter doesn't stop it!
 BUT ENTER DOESN'T STOP IT!
 ```
 
 A problem: text can contain many lines, and the program won't know when to stop.
 
 --
 
 We can terminate the program with Ctrl+C, but it actually expects an .highlight[end of input].
 
 We can signal end of input with Ctrl+D on an empty line (or pressing it twice).
 
 --
 
 .exercise[
   What happens if we press Ctrl+D while back at the shell prompt?
 ]
 
 ---
 
 # Redirecting input
 
 What if we want to use a file as an input in a command that doesn't accept files as arguments,
   we need instruct the shell to use the file as the program's standard input:
 
 .center[
   ![](assets/stdin_redirect.svg)
 ]
 
 This is done with adding .highlight[`< FILE`] to the command.
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' < errors
 LS: CANNOT ACCESS ,: NO SUCH FILE OR DIRECTORY
 ```
 
 ---
 
 # Redirects
 
 .exercise[
 1. Combine input and output redirection to save the output of last `tr` command into `errors.uppercase`
 
 2. Use `cat` to verify the saved output.
 ]
 
 ---
 
 # Pipelines
 
 We have shown how to save outputs to a file, and further process files as inputs.
 
 Sometimes, we don't need to save this intermediate representation.
 In that case, we can directly connect the output of one program to the input of another
 with .highlight[pipes].
 
 To do so, separate two commands with .highlight[`|`]:
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 ---
 
 # Pipelines
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 
 Given this command, shell starts two processes in parallel and ties their respective output and input together.
 
 Standard input/output is connected at the ends of the chain:
 
 .center[
   ![](assets/pipe.svg)
 ]
 
 --
 
 Such .highlight[pipelines] can be longer than two commands, and can be combined with file redirects.
 
 ---
 
 # Filtering output
 
 One extremely useful command used in pipes is `grep`.
 
 It allows to search for text patterns. Example:
 
 ```
 user@host:~/scits-training/io $ ls .
 date.log
 errors
 errors.uppercase
 listing
 user@host:~/scits-training/io $ ls . | grep log
 date.log
 ```
 
 `grep` is versatile:
 
 * Can be used with .highlight[regular expression] patterns
 * Can search for non-matching lines (with `-v`)
 * Can search in files
 * Can print where the match happened in a file
 
 See `man grep` or Google for more examples.
 
 ---
 
 # Pipelines and errors
 
 You will notice that all errors are still output normally:
 
 ```
 user@host:~/scits-training/io $ ls . , | tr 'a-z' 'A-Z'
 ls: cannot access ,: No such file or directory
 .:
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 
 --
 
 As before, errors are not normally redirected, and collected from all processes in the pipe:
 
 .center[
   ![](assets/pipe_err.svg)
 ]
 
 ---
 
 # Background jobs
 
 Recall that, when running a command, the shell waits until it is terminated: all input goes to the program (or nowhere).
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 Sometimes, we don't need to wait until the program terminates – we actually want it running in background.
 
 ---
 
 # Background jobs
 
 If you specify .highlight[`&`] at the end of the command, the shell will start it, but keep control of STDIN:
 
 .center[
   ![](assets/job_stdio.svg)
 ]
 
 Instead of a .highlight[foreground] process, it becomes a .highlight[background] job.
 
 --
 
 You are immediately returned to the shell and can run other commands while the job executes.
 
 Note that both the shell and the background job are connected to STDOUT.
 Redirect output to prevent mix-ups.
 
 ---
 
 # Background jobs
 
 Compare:
 
 ```
 user@host:~/scits-training/io $ sleep 3
 user@host:~/scits-training/io $ sleep 3 &
 [1] 12231
 user@host:~/scits-training/io $
 ```
 
 Here, `[1]` is the .highlight[job number], and `12231` is the .highlight[process ID], or PID.
 
 After 3 seconds and when another command finishes (you can just press Enter for an empty command),
 you'll be informed that the job terminated:
 
 ```
 user@host:~/scits-training/io $
 [1]+  Done                    sleep 3
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Listing jobs
 
 You can list running background jobs with .highlight[`jobs`]:
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12232
 user@host:~/scits-training/io $ sleep 0 &
 [2] 12233
 user@host:~/scits-training/io $ jobs
 [1]-  Running                 sleep 100 &
 [2]+  Done                    sleep 0
 ```
 
 ---
 
 # Terminating jobs
 
 You can forcibly .highlight[terminate] a job with the .highlight[`kill`] command,
   which accepts either PID or job ID (with `%`):
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12234
 user@host:~/scits-training/io $ kill 12234
 user@host:~/scits-training/io $ jobs
 [1]+  Terminated              sleep 100
 
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12235
 user@host:~/scits-training/io $ kill %1
 ```
 
 You can search for more process IDs to terminate with .highlight[`ps ax`], in case something is misbehaving.
 
 ---
 
 # Stopped jobs
 
 Background jobs have nothing connected to their standard input.
 
 If a background job cannot continue without user input, it will .highlight[stop],
 which the shell will signal to you:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' &
 [1] 12236
 user@host:~/scits-training/io $
 [1]+  Stopped                 tr /a-z/ /A-Z/
 user@host:~/scits-training/io $
 ```
 
 --
 
 You can bring a job to .highlight[foreground] to pass STDIN from the shell
 to the running job with `fg` (or `fg %N` for a specific job number):
 
 ```
 user@host:~/scits-training/io $ fg
 tr /a-z/ /A-Z/
 You are now talking to the job
 YOU ARE NOW TALKING TO THE JOB
 ```
 
 ---
 
 # Stopping and resuming programs
 
 You can stop most currently-running programs with `Ctrl`+`Z`:
 
 ```
 user@host:~/scits-training/io $ sleep 100
 ^Z
 [1]+  Stopped                 sleep 100
 user@host:~/scits-training/io $
 ```
 
 --
 
 From there, you can use `fg` to resume normal execution of the program,
   or use .highlight[`bg`] to let it continue to run in the background.
 
 ```
 user@host:~/scits-training/io $ bg
 [1]+ sleep 100 &
 user@host:~/scits-training/io $ jobs
 [1]+  Running                 sleep 100 &
 ```
 
 ---
 
 # Background jobs are fragile
 
 What will happen if you start a background job, and then close the terminal?
 
 --
 
 Closing the terminal (or disconnecting the SSH session) kills the shell you were talking to.
 Since the job was a .highlight[child process] of that shell, it will also be killed.
 
 --
 
 A minor inconvenience if you're working on your own machine (you can just leave the terminal open),
 but a much bigger problem with remote connections.
 
 If the connection is broken, the shell is also terminated along with all processes launched from it.
 
 How to protect against it?
 
 ---
 
 # `screen`
 
 To protect your session, you can use .highlight[`screen`].
 
 `screen` starts a new shell that exists independently of your current one.
 
 Even if the current shell dies (e.g. because you disconnected),
 the shell running in `screen` will continue together with all its child processes.
 
 --
 
 Starting a new `screen` session is simple:
 
 ```
 user@host:~/scits-training/io $ screen
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Reattaching to `screen`
 
 Now suppose your connection was terminated.
 
 Close the terminal where it is running to simulate that, then log in again.
 
 --
 
 You can use .highlight[`screen -ls`] to list active sessions:
 
 ```
 user@host:~ $ screen -ls
 There is a screen on:
         13383.pts-2.host        (11/09/17 03:02:23)     (Detached)
 1 Socket in /var/run/screen/S-user.
 ```
 
 --
 
 You can attach to a screen session (possibly detaching it first, if it's being used somewhere)
   with .highlight[`-dR`] (for .highlight[detach, reattach])
 
 ```
 user@host:~ $ screen -dR
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Controlling `screen`
 
 `screen` can be used for other things, such as having multiple parallel shell sessions open.
 
 Controlling screen consists of pressing .highlight[`Ctrl`+`A`], then a screen-specific command.
 
 For example,
 
 * .highlight[`c`] will .highlight[create] a new shell within screen
 * .highlight[`n`] will switch to the .highlight[next] shell
 * .highlight[`d`] will .highlight[detach] from `screen`, returning you to the original shell
 
 Finally, you can use .highlight[`?`] to access built-in help,
 or use `man screen` for a more detailed manual.
 
 --
 
 Another popular alternative to `screen` is `tmux`. It will not be covered by this tutorial,
   but is worth looking into.
 
 ---
 
 # Users and groups
 
 Before we discuss permissions, we need to understand users and groups in Linux.
 
 A .highlight[user] is a unit of access control; it has a set of credentials to access the system
   and .highlight[owns] some files on it.
 
 A .highlight[group] is a collection of users to facilitate shared access to resources.
   A user can belong to many groups but one group is considered primary.
 
 You can use `id` to check your user and groups:
 
 ```
 akashev@submit01:~ $ id
 uid=7265(akashev) gid=1109(math) groups=1109(math),902(l_gaussian)
 ```
 
 Here, `akashev` is my user, `math` is my primary group and `l_gaussian` is another group I belong to.
 
 ---
 
 # Permissions: `rwx`
 
 Each file and directory in UNIX filesystems has 3 permissions (for a particular user).
 
 Regular files:
 * .highlight[`r`], or .highlight[Read], means that you can read the contents of a file.
 * .highlight[`w`], or .highlight[Write], means that you can modify the file.
 * .highlight[`x`], or .highlight[eXecute], means that the file may be launched as a program.
 
 Directories:
 * .highlight[`r`] means that you can read the list of files within the directory.
 * .highlight[`w`] means that you can add or delete files from the directory.
 * .highlight[`x`] means you can .highlight[traverse] the folder: enter it with `cd` and read the contents of its files.
 
 ---
 
 # Inspecting permissions
 
 Try running `ls -la` to see permissions on files and folders:
 
 ```
 $ ls -la
 total 20
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 drwxrwxr-x 6 user group 4096 Sep 10 23:06 ..
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 -rw-rw-r-- 1 user group   47 Sep 11 00:50 errors
 -rw-rw-r-- 1 user group   30 Sep 11 01:09 listing
 ```
 
 We're interested in the first column: the cryptic `drwxrwxr-x` and `-rw-rwr--`,
 which are called .highlight[mode].
 
 --
 
 * The first character denotes the .highlight[file type].
   * `-` means "regular file".
   * `d` means "directory".
 * The rest is divided in groups of three:
   * Access for the owner
   * Access for the group
   * Access for everyone else
 
 ---
 
 # File ownership
 
 ```
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 Each file in a UNIX filesystem has an .highlight[owner] and a .highlight[group] attached.
 
 In the example above, `user` is the owner and `group` is the designated group.
 
 Note that the user .highlight[doesn't have to be in the assigned group].
 
 ---
 
 # Effective permissions
 
 ```
 -rwxr-x--- 1 user group   90 Sep 10 23:08 script
 ```
 
 To determine which permissions apply, the following is checked:
 
 * If the user is the owner, the first set applies (`rwx`, full permissions)
 * If the user is in the designated group, the second set applies (`r-x`, so cannot write)
 * For all other users, the third set applies (`---`, so cannot do anything)
 
 --
 
 A special user, .highlight[superuser] (normally called .highlight[root]),
   can completely disregard permissions and do anything to any file on the system.
 
 ---
 
 # Permissions: first match applies
 
 Note that the system does not apply "best" permissions – only the first set that matches.
 
 Let's reverse the situation:
 
 ```
 ----r-xrwx 1 user group   90 Sep 10 23:08 script
 ```
 
 For this file, the owner cannot do anything to the file, anyone in `group` cannot modify it,
   but everyone else has full permissions.
 
 Note: the owner can always change a file's permissions.
 
 ---
 
 # Modifying permissions
 
 To modify a file's permissions, use .highlight[`chmod CHANGES FILE`]
 
 .mnemonic[
   `chmod` stands for .highlight[change mode].
 ]
 
 Possible changes:
 * .highlight[`+r`], .highlight[`+w`], .highlight[`+x` add] permissions.
   Can combine: `+rw`
 * .highlight[`-r` removes] permissions.
 * .highlight[`=r`] sets pemissions to .highlight[exactly] `r--`.
 * Prefix .highlight[`u`] changes permissions for the .highlight[user], e.g. `u+r`.
 * Prefix .highlight[`g`] changes permissions for the .highlight[group], e.g. `g+rw`.
 * Prefix .highlight[`o`] changes permissions for .highlight[others], e.g. `o-w`.
 * Prefix .highlight[`a`] or no prefix changes permissions for .highlight[all] three sets.
 * An .highlight[octal number] (e.g. 750) sets permissions to a specific configuration
 (in this case, `rwxr-x---`).
 
 ---
 
 # Modifying permissions
 
 Several changes can be applied at once, separated by commas:
 
 ```
 user@host:~/scits-training/io $ ls -la date.log
 -rwxrw-r-- 1 user group   90 Sep 10 23:08 date.log
 
 user@host:~/scits-training/io $ chmod u+x,g=rx,o-r date.log
 
 user@host:~/scits-training/io $ ls -la date.log
 -rwxr-x--- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 .exercise[
 Modify permissions on the file to be `r-xr--rw-`
 ]
 
 ---
 
 # Changing ownership
 
 Similarly to `chmod`, the .highlight[`chown`] command allows changing a file's owner and group.
 
 * `chown USER FILE` changes the owner
 * `chown :GROUP FILE` changes the group
 * `chown USER:GROUP FILE` changes both
 
 Note: once the owner is changed, the old owner no longer can modify access to the file.
 
 .highlight[For this reason, only administrators can change the file owner, or assign a group the owner is not part of.]
 
 .exercise[
 Use `groups` to list groups you belong to.
 
 Change a file's group to one of them, and then back to the original one.
 ]
 
 ---
 
 # Shell scripting
 
 Shell is not just an interface to launch other programs;
 it comes with its own scripting language to automate complex tasks.
 
 You can have variables, loops, conditionals – a full-featured programming language.
 
 We will only show the very basics.
 
 .exercise[
   Navigate to `~/scits-training/scripts` and open `boom.sh` in your favourite editor (nano, vim)
 ]
 
 ---
 
 layout: true
 
 # Shell scripting
 
 ```bash
 #!/bin/bash
 
 # I hope you get the reference
 echo "Someone set up us the bomb."
 for i in {5..1}
 do
   echo "$i.."
   sleep 1
 done
 explosion="Boom!"
 echo $explosion
 ```
 
 ---
 
 The first line of the script is special:
 ```bash
 #!/bin/bash
 ```
 
 It's called a .highlight["shebang"] (for .highlight[shell] and "!" .highlight[bang]).
 
 It tells the shell what to execute the rest of the script with.
 Since we're writing a `bash` shell script, we put there the path to `/bin/bash` itself.
 
 ---
 
 Other lines starting with .highlight[`#`] are .highlight[comments]
 ```bash
 # I hope you get the reference
 ```
 
 They are ignored by `bash` and are used to leave notes to yourself or others.
 
 ---
 
 .highlight[`echo`] command outputs its arguments to STDIN.
 
 ```bash
 echo "Someone set up us the bomb."
 ```
 
 Quotes are used to make text with spaces in it a single argument; here, they are optional.
 
 ---
 
 .highlight[`for`] designates a .highlight[loop]: a .highlight[variable `i`] will change from 5 to 1.
 
 ```bash
 for i in {5..1}
 do
   # something
 done
 ```
 
 The code in `# something` will repeat with `i` as 5, 4, 3, 2 and 1.
 
 `do` and `done` delimit the bounds of the loop.
 
 ---
 
 One can use the variable in expressions .highlight[prefixed by `$`], i.e. .highlight[`$i`]:
 
 ```bash
 echo "$i.."
 ```
 
 If there is ambiguity as to where a variable name ends, use braces: .highlight[`${i}`],
   e.g. `"Sample ${i}A"` for "Sample 1A", etc.
 
 ---
 
 Variables can also be simply .highlight[assigned to]:
 
 ```bash
 explosion="Boom!"
 echo $explosion
 ```
 
 The lack of spaces around `=` is .highlight[significant].  
 Otherwise Bash will try to execute `explosion` as a command.
 
 ---
 
 layout: false
 
 # Running a script
 
 OK, suppose we wrote the above script. How to execute it?
 
 1. We need to make sure that it's allowed to execute:
 
   ```
   user@host:~/scits-training/scripts $ chmod +x boom.sh
   ```
 
 2. For security reasons, the current directory is not automatically considered when
   starting other programs. We need to explicitly refer to it:
 
   ```
   user@host:~/scits-training/scripts $ ./boom.sh
   ```
 
 .exercise[
 1. Execute the script, saving its output to a file.
 
 2. Modify the script to count down from 10.
 ]
 
 ---
 
 # Scripting, take two
 
 The next script you will type out yourselves.
 
 Open `beer.sh` in your favourite editor.
 
 We'll write a simple script to determine if a user is old enough to drink beer.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 ```
 
 Any bash script should start with an appropriate shebang.
 
 We want to ask the user for his/her age; we can use the .highlight[`read`] command.
 
 ```bash
 # -n prevents a line break, and note the extra space
 echo -n "What's your age? "
 read age
 ```
 
 This will display a promt for the user and wait for input.
 The result is then stored in the variable `$age`.
 
 For simplicity, we will not check that the input is indeed a valid number.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 ```
 
 We need to make a decision based on age; we need an if-then-else construct.
 
 ```bash
 if [ $age -lt 16 ]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .highlight[`fi`] here is `if` reversed, to close the .highlight[`if`] statement.
 
 Conditionals in bash are a bit clunky, but .highlight[`-lt`] here stands for .highlight[less than].
   Again, the whitespace here is .highlight[significant].
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 if [ $age -lt 16 ]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .exercise[
 1. Save this script to `beer.sh`.
 
 2. Change the file's mode to allow execution.
 
 3. Test the script with different values.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's add a little personal touch.
 
 `whoami` is a command that returns the username. Let's edit `beer.sh` to use it:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
 else
   echo "$(whoami), you're old enough, have a beer!"
 fi
 ```
 
 .highlight[`$(something)`] allows you to execute a command and substitute the result within another command.
 
 .exercise[
 Test the new additions.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's read the age from the command line arguments.
 
 `bash` automatically populates .highlight[`$0`] with the .highlight[name of the executable],
 and .highlight[`$1`], .highlight[`$2`] and so on with .highlight[arguments].
 
 Let's use `$1` as age if it's defined:
 
 ```bash
 if [ $1 ]
 then
   age=$1
 else
   echo -n "What's your age? "
   read age
 fi
 ```
 
 .exercise[
 Test that `./beer.sh` now automatically gets the age from its first argument, and still asks if no argument is provided.
 ]
 
 ---
 
 # Return values
 
 Whenever a program terminates, it returns a single integer to the shell that called it;
   it's called the .highlight[return value].
 
 By convention:
 * .highlight[0] means "no error".
 * any .highlight[non-zero value] means "some kind of error".
 
 Let's return appropriate values:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
   exit 1
 else
   echo "$(whoami), you're old enough, have a beer!"
   exit 0
 fi
 ```
 
 ---
 
 # Chaining commands
 
 You can chain commands in shell with .highlight[`;`] or .highlight[`&&`].
 
 .highlight[`;`] will execute commands one by one, regardless of errors.
 
 ```
 $ command1; command2
 ```
 
 .highlight[`&&`] will only execute the next command only if the previous one returned 0, i.e. finished without errors.
 ```
 $ command1 && command2
 ```
 
 .exercise[
 1. Apply the return value changes to `beer.sh`
 
 2. Test it with `./beer.sh && echo 'Cheers!'`
 ]
 
 ---
 
 # Environment variables
 
 Your profile files can set various .highlight[environment variables]:
 snippets of data inherited by programs running from shell.
 
 You can see your current environment variables with:
 
 ```
 $ env | less
 ```
 
 --
 
 Some programs rely on environmental variables to change their behavior. Example:
 
 ```bash
 # Will replace the default editor with vim in some commands
 export EDITOR=vim
 ```
 
 ---
 
 # Environment variables
 
 Environment variables work similarly to variables in a script, except for the extra
   command .highlight[export], which propagates this variable to child processes.
 
 By convention, environment variables are UPPERCASE.
 
 --
 
 .exercise[
 1. Set an ordinary varable A (`A='Hello'`) and environment variable B (`export B='World'`).
 
 2. Use `echo $A` and `echo $B` to display them.
 
 3. Add `echo $A` and `echo $B` to a script and execute it. What changes?
 ]
 
 ---
 
 # `$PATH` variable
 
 An important variable is `$PATH`.
 
 It's a colon-separated list of directories which are searched when you try to run a program by name.
 
 Notably, the current directory is not in `$PATH`.
 
 If you have created some own scripts/programs and want them to be available by name from anywhere,
   you can put them in a folder (e.g. `~/bin`) and add it to `$PATH`:
 
 ```bash
 export PATH="$PATH:$HOME/bin"
 ```
 
 ---
 
 # `$PATH` variable, example
 
 .exercise[
 1. Try running `beer.sh` directly y name. It fails.
 
 2. Add the folder that holds it to `$PATH`:
 
 ```bash
 export PATH="$PATH:~/scits-training/scripts"
 ```
 
 3. Try running `beer.sh` now.
 
 4. Try going somewhere else (`cd ~`) and run it.
 ]
 
 ---
 
 # Aliases
 
 If you use a certain command often, you can define a short name for it.
 
 For example, if you want a shorter name for `ls -lh` because you always want to see human-readable sizes,
 you can make an alias:
 
 ```
 $ alias lh="ls -lh"
 $ lh
 total 26M
 -rw-r--r-- 1 user group 25M Sep 11 07:22 big_file
 -rw-r--r-- 1 user group 735 Sep 11 07:22 description
 -rw-r--r-- 1 user group   0 Sep 11 07:22 empty_file
 -rw-r--r-- 1 user group 551 Sep 11 07:22 naming
 drwxr-xr-x 0 user group 512 Sep 11 07:22 subfolder
 ```
 
 ---
 
 # Making customizations permanent
 
 To make above tweaks permanent, they need to be added either to `.bash_profile` or `.bashrc`.
 
 Then they will apply on each opened shell.
 
 * `.bash_profile` is .highlight[sourced] at most once. Put things there that shouldn't be called multiple times.
 
 * `.bashrc` is sourced almost every time `bash` is called, except for initial SSH shell.
   To be safe, you can "include" `.bashrc` into `.bash_profile` like this:
 
 ```bash
 # In .bash_profile
 # -f tests that file exists
 # source executes commands in the current shell
 if [ -f ~/.bashrc ]; then
   source ~/.bashrc
 fi
 ```
 
 ---
 
 class: center, middle
 
 # Extra credits
 
 ---
 
 # Searching through history
 
 There's a way to quickly search through previous commands.
 
 `Ctrl`+`R` opens "reverse search" mode. Enter some pattern and the closest command in history that matches will be shown.
 
 To look into older commands, press `Ctrl`+`R` again, or `Ctrl`+`C` to abort.
 
 ---
 
 # Custom shell prompt
 
 The variable `$PS1` contains the format template for your shell prompt.
 
 Throughout this training, you saw the following prompt:
 
 ```
 user@host:~ $ 
 ```
 
 You can customize it! For example:
 ```
 user@host:~ $ export PS1="[\t] \u@\h:\w\\n\\$ "
 
 [16:40:00] user@host:~
 $ 
 ```
 
 Want to control that precisely? Want to add color?
 
 There's [a guide](https://www.digitalocean.com/community/tutorials/how-to-customize-your-bash-prompt-on-a-linux-vps) for that.
 
 ---
 
 # Finding files
 
 The `find PATH` command looks through the filesystem at PATH to find files.
 
 One can then filter the output with `grep`, or use `find`'s own keys for sophisticated filtering.
 
 ```
 user@host:~ $ find ~/scits-training -name '*.sh'
 /home/user/scits-training/scripts/boom.sh
 /home/user/scits-training/scripts/beer.sh
 ```
 
 ---
 
 # `xargs`
 
 The `xargs` command can be used to convert output into arguments of another command.
 
 `xargs COMMAND` will take input and pass it as separate arguments after `COMMAND`:
 
 ```
 user@host:~ $ find ~/scits-training -name '*.sh' | xargs cat
 [contents of both .sh files]
 ```
 
 This is equivalent to 
 ```
 user@host:~ $ cat /home/user/scits-training/scripts/boom.sh \
 /home/user/scits-training/scripts/beer.sh
 ```
 
 ---
 
 # Public key authentication: theory
 
 It can be useful to use key authentication instead of standard password authentication.
 
 * Far more secure — suitable for internet-facing computers.
 * May be required in cloud environments to set new VMs.
 * Allows passwordless authentication for more convenience.
 
 --
 
 It is based on modern cryptography and consists of pairs of keys: .highlight[public],
   which you can give to others, and .highlight[private], that you keep yourself
   (preferably encrypted with a .highlight[passphrase]).
 
 Having the private key allows you to prove that you own the keypair to anyone having
   your public key, without disclosing the private key itself.
 
 ---
 
 # Public key authentication: workflow
 
 When setting up public key authentication on Linux, here's the workflow:
 
 1. You generate a keypair: private key and public key files.
 
   On Linux/Mac, `ssh-keygen` is used. On Windows, PuTTYgen can be used.
 
 2. You copy the public to the remote system.
 
 3. You connect, instructing SSH to use the private key.
 
   If it's encrypted, you're asked for the passphrase (and may be cached in an SSH agent after that).
 
 ---
 
 # Public key authentication: workflow
 
 On most Linux systems, it is sufficient to have your public key in `~/.ssh/authorized_keys` file.
 
 To copy the keypair to the remote system, `ssh-copy-id` script can be used.
 
 --
 
 Otherwise, you need to create that file/folder yourself, and make sure they have proper permissions.
 
 ```
 user@remote:~$ ls -l ~/.ssh
 total 12
 drwx------  2 user user 4096 Sep 13  2017 .
 drwxr-xr-x 13 user user 4096 Sep  2 21:47 ..
 -rw-------  1 user user 1159 Jan 21  2018 authorized_keys
 user@remote:~$ cat .ssh/authorized_keys
 ssh-rsa AAAAB3NzaC1yc2EAAA[...]j6aKfAUoXOE= some comment
 ```
 
 --
 
 To use a key, one can use `-i PRIVATE_KEY_FILE` flag for `ssh`, or Pageant on Windows.

- \ No newline at end of file diff --git a/linux_intro.html b/linux_intro.html index a7b2be9..f5ed549 100644 --- a/linux_intro.html +++ b/linux_intro.html @@ -1,3617 +1,3615 @@ Introduction to Linux -

class: center, middle
 
 # Introduction to Linux
 
 ## Part I
 
 `https://goo.gl/Vg3iXW`
 
 ---
 
 # Agenda
 
 1. What is Linux?
 2. Linux interface: GUI vs CLI
 3. Connecting to a remote Linux system
 4. Linux directory structure
 5. Moving and looking around
 6. Reading and writing files
 7. Organizing files and folders
 8. Moving data from/to a remote Linux system
 
 ---
 
 class: center, middle
 
 # What is Linux?
 
 ![](assets/Tux.png)
 
 ---
 
 class: center
 
 # What is Linux?
 
 --
 
 <br>
 
 The most common answer you'll hear is:
 
 ## "Linux is an operating system"
 
 --
 
 <br>
 
 But what does this mean?
 
 ---
 
 class: center
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td></td>
       <td></td>
     </tr>
     <tr>
       <td>System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td>System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td></td>
     </tr>
     <tr>
       <td>Kernel</td>
       <td></td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td class="bluebg">System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td rowspan="2"><strong>Operating system</strong><br>Windows, Linux, MacOS, Android, ...</td>
     </tr>
     <tr>
       <td class="bluebg">Kernel</td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 class: center
 count: false
 
 # Operating systems
 
 <br>
 
 <div class="center">
   <table class="stack">
     <tr>
       <td>User software</td>
       <td></td>
     </tr>
     <tr>
       <td class="bluebg">System software<br><em>(shell, utilities, libraries, ...)</em></td>
       <td rowspan="2"><strong>Linux</strong></td>
     </tr>
     <tr>
       <td class="bluebg">Kernel</td>
     </tr>
     <tr>
       <td>Hardware + Firmware</td>
       <td></td>
     </tr>  
   </table>
 </div>
 
 ## In practice, we call this part "Linux"
 
 ---
 
 # Linux? Wait, I also heard "UNIX"?
 
 UNIX is the name of an operating system from 1970 that pioneered concepts that
   will form the basis of Linux (and other OSes) today.
 
 More importantly, it introduced a set of conventions that its descendents follow.
   A system that follows them is called "UNIX-like".
 
 Most of what you learn here will easily transfer to other UNIX-like OSes (e.g. macOS).
 
 ---
 
 ![:scale 100%](assets/Unix_timeline.en.svg)
 
 ---
 
 class: center, two-columns
 
 # User Interface
 
 GUI                 | CLI
 ------------------- | -------------
 Graphical Interface | Command Line
 ![:scale 80%](assets/ubuntu-desktop.png) | ![:scale 100%](assets/terminal.png)
                     | Some synonyms:
                     | "Shell", "Terminal", "TTY"
 
 ---
 
 # Command Line Interface
 
 ```
 user@host:~ $ cowsay "Command Line Interface"
  ________________________
 < Command Line Interface >
  ------------------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||
 ```
 
 --
 
 * Input, output, and commands are **text**.
 
 --
 
 * Easy on the computer: can run on any hardware.
 
 --
 
 * Network-friendly: a few bytes of text vs realtime stream of images / GUI updates => tool of choice for remote access.
 
 --
 
 * Scripting/automation-friendly: text is easier to manipulate.
 
 --
 
 * Expert-friendly, but _beginner-unfriendly._
 
 ---
 
 # Connecting to a remote Linux system
 
 The standard tool to connect to a remote system is `ssh`.
 
 .acronym[
 SSH: .highlight[S]ecure .highlight[Sh]ell
 ]
 
 It securely connects you to a remote system.  
 Communication is encrypted, both parties are authenticated.
 
 First, you will need to log in to the system.
 
 If your credentials are accepted, it creates a new shell for you.
 
 It is then displayed on your screen and controlled by your keyboard,
 relayed over the network.
 
 ---
 
 # Connecting from MacOS / Linux:
 
 Good news: you already have a terminal and `ssh` of your own!
 
 --
 
 First, open the terminal:
 
 * For MacOS, it's accessible from Launchpad, Utilities.
 * For Linux GUI, usually look for a program called Terminal.
 
 --
 
 Then, you need to input the command to connect to a remote host:
 
 ```
 local.user@local:~ $ ssh user@remote
 
 [..some mutual* authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 # Connecting from Windows:
 
 You will need an SSH client. Standard one: PuTTY
 
 .center[.middle[
 ![:scale 20%](assets/putty-logo.jpg) ![:scale 50%](assets/putty-main.jpg)
 ]]
 
 Download the appropriate installer: https://goo.gl/pHFReU
 
 ---
 
 # Connecting from Windows:
 
 * Make sure "Connection type: SSH" is selected.
 * Put the remote's host name / IP in the form.
 * Select "Open"
 
 A terminal window will open..
 
 ```
 [..some mutual* authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 background-image: url(assets/tofu.svg)
 background-position: top right
 background-size: 40%
 
 # Mutual authentication?
 
 (and what's up with this side picture?)
 
 ---
 
 background-image: url(assets/tofu.svg)
 background-position: top right
 background-size: 40%
 
 # Mutual authentication?
 
 SSH authenticates both parties:
 
 * Client to server
   * Username + password
   * Username + cryptographic key
   * Something else!
 * Server to client
   * The server has a cryptographic key to prove its identity
 
 --
 
 The first time you connect, you need to explicitly say you trust the (previously unknown) server.
 
 On subsequent connections, SSH will verify that you are still connecting to a server with the same key,
 and will warn you before login credentials are transmitted if you aren't.
 
 --
 
 This is called TOFU (.highlight[T]rust .highlight[O]n .highlight[F]irst .highlight[U]se).
 
 ---
 
 layout: true
 
 # Mutual authentication
 
 So, the first time you connect to a new server, you should _expect_ a warning you need to confirm:
 
 ---
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)?
 ```
 
 ---
 
 count: false
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)? yes
 ```
 
 ---
 
 count: false
 
 In Linux/MacOS:
 
 ```
 local.user@local:~ $ ssh user@remote
 The authenticity of host 'remote (11.22.33.44)' can't be established.
 ECDSA key fingerprint is SHA256:eQZbiUM4qV6ptjc0fN6/pFglj45qaNlXbLCULCTzSGM.
 Are you sure you want to continue connecting (yes/no)? yes
 Warning: Permanently added 'remote,11.22.33.44' (ECDSA) to the list of known 
 hosts.
 
 [..some client authentication later..]
 
 user@remote:~ $
 ```
 
 ---
 
 In Windows/PuTTY:
 
 .center[
   ![:scale 60%](assets/putty-server-auth.png)
 ]
 
 ---
 
 layout: false
 
 # Hands-on time: Connect to a server
 
 Using your Campus account username/password, use SSH/PuTTY to connect to UBELIX at `submit.unibe.ch`.
 
 ```
 ssh user@submit.unibe.ch
 ```
 
 A reminder, PuTTY can be obtained from https://goo.gl/pHFReU
 
 ---
 
 # Greetings from a shell
 
 ![:scale 100%](assets/greetings_shell.jpg)
 
 ---
 
 # Greetings from a shell
 
 After connecting, you will be greeted with something like this:
 
 ```
 user@remote:~ $
 ```
 
 --
 
 What you see is the interface of the .highlight[shell]: a text-based interface that allows you to launch other programs with commands.
 
 --
 
 .mnemonic[
 It's called a shell .highlight[prompt] since it's .highlight[prompting] you to enter a command.
 ]
 
 The prompt contains a short summary of current state of the shell.
 
 ---
 
 # Anatomy of a prompt
 
 The prompt looks like this:
 
 ```
 user@remote:~ $
 ```
 
 This may vary slightly from system to system, and is fully configurable, but this is the typical form.
 
 --
 
 This form of the prompt answers 3 questions:
 
 * Who are you? .highlight[Username `user`]
 
 * Where are you? .highlight[Hostname `remote`]
 
 * Where in the filesystem are you? .highlight[`~`] (explained later)
 
 --
 
 Terminating the prompt is (traditionally) a .highlight[`$`] character:
 it delimits where your input goes.
 
 ---
 
 layout: true
 
 # Taking command
 
 The shell expects a textual command; most of the time you type the command and press [ENTER] to commit it.
 
 Let's try this (in slow motion)!
 
 ---
 
 ```
 user@remote:~ $
 
 ⠀
 ```
 
 ---
 
 count: false
 
 ```
 user@remote:~ $ whoami
 
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits
 
 ---
 
 count: false
 
 ```
 user@remote:~ $ whoami
 
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits
 2. Pressing [ENTER]. The shell will process the command (launch the program `whoami`)
 
 ---
 
 count: false
 
 ```
 user@remote:~ $ whoami
 user
 ⠀
 ```
 
 1. Typing in "whoami" as the shell waits.
 2. Pressing [ENTER]. The shell will process the command — launch the program `whoami`.
 3. The program will take over input/output — in this case, it will output your username).
 
 ---
 
 count: false
 
 ```
 user@remote:~ $ whoami
 user
 user@remote:~ $ ⠀
 ```
 
 1. Typing in "whoami" as the shell waits.
 2. Pressing [ENTER]. The shell will process the command — launch the program `whoami`.
 3. The program will take over input/output — in this case, it will output your username).
 4. The program terminates, and control returns to the shell; it shows a new prompt.
 
 ---
 
 layout: false
 
 # Try it!
 
 Here's a few commands for you to try:
 
 ```
 whoami
 
 echo Hello!
 
 pwd
 
 ls -l
 
 date
 
 sleep 3
 
 clear
 
 history 5
 ```
 
 Each should do something and return you to the shell prompt.  
 Can you guess what they do?
 
 Note that you can use up/down arrows to access/repeat previous commands.
 
 ---
 
 # Safety first, or emergency exits!
 
 So far every command we encountered automatically returned control back to the shell.
 
 But what if a program is stuck, or expecting some input and you're not sure what to do?
 
 Typical shortcuts to stop / quit a program:
 
 * `Ctrl` + `C` (also called .highlight[interrupt])
 * `Esc` (from ".highlight[escape]")
 * `q` (from ".highlight[q]uit")
 * `Ctrl` + `D` (.highlight[end of input], in case a program is waiting)
 
 If you try those, usually you'll either exit the program or get some hint on how to do it.
 
 `Ctrl` is sometimes denoted as `^`, e.g. `^C` for `Ctrl`+`C`.
 
 ---
 
 exclude: true
 
 # Anatomy of a shell command
 
 The shell expects input. What does it (typically) look like?
 
 ```
 user@remote:~ $ program -f --option abc 123
 ```
 
 Here, `program` is the .highlight[command] being executed, and the rest is the list of its .highlight[arguments]:
 
 1. `-f`
 2. `--option`
 3. `abc`
 4. `123`
 
 Arguments that start with `-` or `--` are often called .highlight[flags] or .highlight[switches] and traditionally change
 some options of the command.
 
 ---
 
 # The Working Directory
 
 One of the commands you executed, `pwd`, printed a directory path (of your home directory, by default):
 
 ```
 akashev@submit01:~ $ pwd
 /home/ubelix/math/akashev
 ```
 
 --
 
 .mnemonic[
   `pwd` stands for .highlight[P]rint .highlight[W]orking .highlight[D]irectory
 ]
 
 --
 
 Whenever you use the shell, there is a concept of the current (or "working") directory.
 This affects how commands search for files and how they interpret paths.
 
 Think of it as of "where" you are: if a server is a building you're in, a working directory
   is the room you're in within that building.
 
 
 ---
 
 # The Working Directory
 
 One of the commands you executed, `pwd`, printed a directory path (of your home directory, by default):
 
 ```
 akashev@submit01:~ $ pwd
 /home/ubelix/math/akashev
 ```
 
 Usually, this information is printed in the shell prompt itself, to remind you of the current state.
 
 --
 
 In this example it's `~`, which represents the .highlight[home directory].
 
 --
 
 Here's how it would look if you were somewhere else, for example in `/var/log`:
 
 ```
 akashev@submit01:/var/log $
 ```
 
 ---
 
 # UNIX directory structure
 
 If you're reading this tutorial, you likely already know that files are normally organized
   into nested "directories" (or "folders"). For example, on Windows you may have such a path:
 
 ```
 C:\folder\subfolder\file
 ```
 
 On Linux, paths looks similarly:
 
 ```
 /home/user/folder/subfolder/file
 ```
 
 ---
 
 # UNIX directory structure
 
 ```
 /folder/subfolder/file
 ```
 
 * a file .highlight[`file`]
 * inside a directory .highlight[`subfolder`]
 * which is inside a directory .highlight[`folder`]
 * which itself is inside the .highlight[root directory] `/`
 
 ![](assets/file-tree.svg)
 
 ---
 
 # The `/` as directory separator
 
 Forward slashes (`/`) separate the folders in the path.  
 Using multiple is valid, so the following is the same file:
 
 ```
 /home/user/folder/subfolder/file
 ///home/user///folder/subfolder//file
 ```
 
 --
 
 A path to a regular file never ends in `/`, e.g. this is not valid:
 
 ```
 /home/user/folder/subfolder/file/
 ```
 
 --
 
 Directories can be referred to with or without the final `/`:
 
 ```
 /home/user/folder/subfolder
 /home/user/folder/subfolder/
 ```
 
 --
 
 Root directory is special: `/` is its only name.
 
 ---
 
 # Absolute and relative paths
 
 If a path starts with `/`, it's an .highlight[absolute] path that starts at root:
 
 ```
 /home/user/folder/subfolder/file
 ```
 
 --
 
 If it does not, then it's a .highlight[relative] path
 that starts at the current working directory instead of `/`.
 
 If the current working directory is
 
 ```
 /home/user/folder
 ```
 
 then the following paths point to the same file:
 
 ```
 /home/user/folder/subfolder/file
 subfolder/file
 ```
 
 ---
 
 # Absolute and relative paths
 
 ```
 /home/user/folder/subfolder/file
 subfolder/file
 ```
 
 ![:scale 60%](assets/File-tree-2.svg)
 
 ---
 
 # Special folders `.` and `..`
 
 There are 2 special folders inside each folder: `.` and `..`
 
 * `.` points to the folder itself.
 
 ```
 /home/user/folder/subfolder/./file
 ```
 
 --
 
 * `..` points to one folder "up" in the path. At root, it points to root itself.
 
 ```
 /home/user/another_folder/../folder/file
 /home/../../home/user/folder/file
 ```
 --
 
 It's mostly important for relative paths:
 
 ```
 # From /home/user/another_folder
 ../folder/file
 ```
 
 ---
 
 # Special folders `.` and `..`
 
 From `/home/user/another_folder`
 
 ```
 /home/user/folder/subfolder/file
 .././folder/subfolder/file
 ```
 
 ![:scale 60%](assets/File-tree-3.svg)
 
 ---
 
 # Home directories
 
 Each user has a .highlight[home directory] assigned.
 
 It acts as your default working directory.
 
 By convention, its path usually starts with `/home/`
 and ends with your username:
 
 ```
 /home/<maybe something else>/username
 ```
 
 It's frequently referred to as `~`:
 
 ```
 /home/username/folder/file
 ~/folder/file
 ```
 
 You can even refer to others' home folder with `~username`:
 
 ```
 /home/someone/file
 ~someone/file
 ```
 
 ---
 
 # Quiz time! [1/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /scratch/folder/B
 Target:             /scratch/folder/A/a
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>~/../scratch/folder/A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~userA/../../scratch/folder/A/a</code></td>
     </tr>
     <tr>
       <td class="N"><code>A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>../A/a</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/scratch/./folder/A/a</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Quiz time! [2/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /home/userA/temp
 Target:             ../../userB/folder/file
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>/home/userB/userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/home/userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="N"><code>~/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~/../userB/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~userB/folder/file</code></td>
     </tr>
   </table>
 </div>
 
 ---
 
 # Quiz time! [3/3]
 
 Suppose the following:
 
 ```
 Username:           userA
 Home directory:     /home/userA
 
 Working directory:  /home/userA/folder
 Target:             /home/userA/folder/file
 ```
 
 Which of those paths point to the target? (click to reveal)
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>./file</code></td>
     </tr>
     <tr>
       <td class="N"><code>~/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>~/folder/file</code></td>
     </tr>
     <tr>
       <td class="Y"><code>/home/userA/folder/subfolder/../file</code></td>
     </tr>
   </table>
 </div>
 
 ---
 
 # Preparing for training
 
 Please execute the following command to add the exercises to your home folder:
 
 ```
 $ wget https://scits.math.unibe.ch/script -O - | /bin/bash
 ```
 
 (That's a capital O and spaces are significant)
 
 This should be the only time you don't understand what you're doing;
   and by the end of Part II you should understand it.
 
 Hint: On a Swiss German keyboard, `|` is <kbd>AltGr</kbd> + <kbd>7</kbd>
 
 ---
 
 # Moving around
 
 Now that we know:
 
 * Files and directories are organised in a tree
 * There's a "current"/working directory that we are in
 
 we need to learn to move around in that tree.
 
 --
 
 For that, we need the `cd` command:
 
 ```
 user@remote:~ $ cd scits-training
 user@remote:~/scits-training $ pwd
 /home/username/scits-training
 user@remote:~/scits-training $
 ```
 
 .acronym[
 `cd` stands for ".highlight[C]hange .highlight[D]irectory"
 ]
 
 ---
 
 # Moving around 
 
 The general format of the command is `cd DESTINATION`, where `DESTINATION` is a path
 (relative or absolute) to a directory.
 
 ```
 user@remote:~ $ cd scits-training
 user@remote:~/scits-training $ cd /usr/local/bin
 user@remote:/usr/local/bin $
 ```
 
 --
 
 To go "back up", one uses the special `..` directory:
 
 ```
 user@remote:/var/local/bin $ cd ..
 user@remote:/usr/local $ cd ../..
 user@remote:/ $
 ```
 
 --
 
 To go to your home directory, you can use `~`:
 
 ```
 user@remote:/ $ cd ~
 user@remote:~ $
 ```
 
 ---
 
 # `cd` shortcuts
 
 There are two useful tricks when using `cd`:
 
 "`cd -`" goes back to the previous directory you were in:
 
 ```
 user@remote:~ $ cd -
 user@remote:/ $
 ```
 
 And "`cd`" without arguments goes to your home folder:
 ```
 user@remote:/ $ cd
 user@remote:~ $
 ```
 
 ---
 
 # Tab-completion
 
 This is a good point to introduce a helpful CLI tool: .highlight[tab completion]
 
 When entering a command, you can press the `[Tab]` key to suggest a command, or path,
 based on already entered input.
 
 ```
 user@remote:~ $ cd scits-training/a
 ```
 
 Pressing `[Tab]` now completes the name, since it's the only one that matches the beginning:
 
 ```
 user@remote:~ $ cd scits-training/animals/
 ```
 
 (continues on next slide)
 
 ---
 
 layout: true
 
 # Tab-completion
 
 ```
 user@remote:~ $ cd scits-training/animals/
 ```
 
 Pressing `[Tab]` once again won't change anything, since there are mutiple choices for completion;
 however, if it is pressed again, it shows possibilities:
 
 ```
 user@remote:~ $ cd scits-training/animals/
 Aardvark/ Badger/
 user@remote:~ $ cd scits-training/animals/
 ```
 
 ---
 
 ---
 
 count: false
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/A
 ```
 
 ---
 
 count: false
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/Aardvark/
 ```
 
 ---
 
 count: false
 
 The shell needs to know the next letter to proceed. So, we type only "A" and press `Tab` again:
 
 ```
 user@remote:~ $ cd scits-training/animals/Aardvark/
 user@remote:~/scits-training/animals/Aardvark/ $ 
 ```
 
 ---
 
 layout: false
 
 # Looking around
 
 To look around in a UNIX filesystem, you use the `ls` command:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls
 description  empty_file  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 .mnemonic[
 `ls` stands for .highlight[list]
 ]
 
 This lists the names for contents of the working directory.
 
 --
 
 We can specify another folder to look at:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls ../Badger/
 Arctonyx  Meles  Mellivora  Melogale  Mydaus
 ```
 
 ---
 
 # Looking around (in depth)
 
 To show more information, we can use the `-l` (for .highlight[l]ong) flag:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -l
 total 25640
 -rw-r--r-- 1 username groupname 26214400 Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname      754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname        0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname     4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 Important information from this output:
 
 * `-rw-r--r--` is called the .highlight[mode] (explained in Part II).
   * `d` denotes .highlight[directory] in this example.
   * `rw-r--r--` deals with permissions for the files.
 * `username` and `groupname` are .highlight[owners] of the file.
 * The number after `groupname` is the .highlight[size] (in bytes) of the file.
   * Important: for folders, it's not the size of all contents.
 * The date/time after the size is the .highlight[modification date].
 
 ---
 
 # Looking around (as puny humans)
 
 One can use the flag `-h` (for .highlight[h]uman-readable) for more familiar size units:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -l -h
 total 26M
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 Single-letter flags in commands can often be combined:  
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -lh
 total 26M
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 ---
 
 # Looking around (into hidden corners)
 
 Another often-used flag is `-a` (for .highlight[a]ll): it lists contents with names that start with a dot `.`
 which are normally hidden in UNIX.
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -a
 <span style="color:#3030ff;font-weight:bold">.</span>  <span style="color:#3030ff;font-weight:bold">..</span>  big_file  description  empty_file  .hidden  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 As usual, it can be combined with others:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls -lah
 total 26M
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">.</span>
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">..</span>
 -rw-r--r-- 1 username groupname  25M Aug 28 18:20 big_file
 -rw-r--r-- 1 username groupname  754 Aug 25 17:55 description
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 empty_file
 drwxr-xr-x 2 username groupname 4096 Aug 28 16:52 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 -rw-r--r-- 1 username groupname    0 Aug 28 16:51 .hidden
 ```
 
 ---
 
 
 # Looking around (in orderly fashion)
 
 By default, files are ordered by name.
 
 This behavior can be changed with flags; here are some examples:
 
 * `-r` .highlight[r]everses the sort order.
 * `-S` sorts files by .highlight[s]ize.
 * `-t` sorts files by modification .highlight[t]ime.
 * `-X` sorts files by filename e.highlight[x]tension, e.g. `png` in `image.png`.
 
 As usual, this can be combined with the previous ones.
 
 .exercise[
 List files in `Aardvark` by increasing size.
 ]
 
 ---
 
 # I'm never going to remember this!
 
 .center[
 ![](assets/tar.png)
 ]
 
 ## Good news: you don't have to.
 
 As long as you remember the command's name, you can look up its correct usage from the terminal itself.
 
 .small[Image credit: https://xkcd.com/1168/]
 
 ---
 
 layout: true
 
 # Getting help
 
 Some common methods of getting help:
 
 ---
 
 * Many programs support .highlight[`--help` flag] to print out their usage instructions:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls --help
 Usage: ls [OPTION]... [FILE]...
 List information about the FILEs (the current directory by default).
 Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
 
 Mandatory arguments to long options are mandatory for short options too.
   -a, --all                  do not ignore entries starting with .
 [...]
 ```
 
 ---
 
 * For most programs, you can look up their .highlight[manual file] with `man`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ man ls
 ```
 
 Instead of just outputting the text and returning, you'll enter a mode for showing long files.
 
 Look around using arrow keys and `PgDn`/`PgUp`.
 
 Remember the hints on how to exit (here, it's `q`).
 
 You can search a `man` page for "something" with `/something` and just `/` to go to the next find.
 
 ---
 
 * Some commands are not separate programs, but are .highlight[built into the shell], e.g. `cd`.
   For those, you can use `help`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ help cd
 ```
 
 --
 
 You can see what `help` can help with as well:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ help
 ```
 
 ---
 
 layout: false
 
 # Try out `man`
 
 Try opening the manual for `ls`:
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ man ls
 ```
 
 Reminders:
 
 * You can search a `man` page for "something" with `/something` and `n` to go to the next find.
 
 * To exit, you can use `q`.
 
 .exercise[
 Try searching for the meaning of `-R` flag, and try to use it.
 ]
 
 ---
 
 # Reading files
 
 We know how to look around the filesystem (with `ls`) and how to move around (with `cd`).
 
 However, we still need to access the contents of files.
 
 There are many ways to do that, I'll show a few more common ones.
 
 ---
 
 background-image: url(assets/cat.svg)
 background-position: top 5% right 30%
 background-size: 15%
 
 # Simple file reading
 
 The simplest program to read the file is `cat`
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ ls
 big_file  description  empty_file  naming  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 user@remote:~/scits-training/animals/Aardvark/ $ cat description
 The aardvark (ARD-vark; Orycteropus afer) is a medium-sized, burrowing,
 [...]
 ```
 
 .mnemonic[
 `cat` comes from the word "con.highlight[cat]enate",  
 which means joining things together in a series.
 ]
 
 .exercise[
 What happens if we call `cat` with two filenames?
 
 ```cat description naming```
 ]
 
 ---
 
 background-image: url(assets/cat_long.svg)
 background-position: top 5% right 20%
 background-size: 25%
 
 # File is too long!
 
 Sometimes a file is too long to be comfortably read with `cat`
 
 ```
 user@remote:~/scits-training/animals/Aardvark/ $ cd ../../numbers/
 user@remote:~/scits-training/numbers/ $ cat hundred
 1
 2
 [...]
 99
 100
 ```
 
 A hundred lines is too much to fit into the terminal window.
 
 While you can scroll to look through the output, sometimes files are much longer than that.
 
 We can display only parts of the file, or use a program that allows to navigate a file.
 
 ---
 
 background-image: url(assets/cat_parts.svg)
 background-position: top 5% right 30%
 background-size: 15%
 
 # Parts of a cat?
 
 If a cat is too long, perhaps we only need to look at its beginning (`head`)
 or end (`tail`):
 
 ```
 user@remote:~/scits-training/numbers/ $ head hundred
 1
 [...]
 10
 ```
 
 ```
 user@remote:~/scits-training/numbers/ $ tail hundred
 91
 [...]
 100
 ```
 
 Those commands display the first and last 10 lines of a file, respectively.
 
 .mnemonic[
   Remembering `cat` together with `head` and `tail` may help.
 ]
 
 ---
 
 # Self-help test
 
 Of course, you can look up other options with the self-help methods like `man`.
 
 .exercise[
 Use one of the help methods (`man head` or `head --help`) to learn how to display 5 lines instead of 10 with `head`.
 ]
 
 Hint: it will be a flag that should go before the filename.
 
 --
 
 **Answer:** `-n 5`, `-n5` or `--lines=5`
 
 ```
 user@remote:~/scits-training/numbers/ $ head -n 5 hundred
 1
 2
 3
 4
 5
 ```
 
 ---
 
 # The file is too long, show less
 
 One way to navigate a big file is `less`:
 
 ```
 user@remote:~/scits-training/numbers/ $ less hundred
 ```
 
 You will recognize this interface, since `man` also uses `less`.
 
 Commands to try:
 * .highlight[Arrow keys] to scroll line by line
 * .highlight[`PgUp`] / .highlight[`PgDn`] to scroll screen by screen
 * .highlight[`/something`] to search for "something"
 * .highlight[`n`] to go to next found "something", .highlight[`N`] to go back
 * .highlight[`>`] to go to the end of the file, .highlight[`<`] to go to the beginning
 * .highlight[`h`] to show help
 * .highlight[`q`] to quit
 
 ---
 
 # Modifying files
 
 Besides reading, we need to be able to create and modify files.
 
 There are many editors available, and which one is "best" can lead to [hot debate](https://xkcd.com/378/).
 
 We will mention and briefly explain two editors that are likely to be installed on any system you encounter nowadays.
 
 * `nano`
 * `vim`
 
 ---
 
 # `nano`
 
 ```
 user@remote:~/scits-training/numbers/ $ nano hundred
 
 <span style="background:white;color:black;">GNU nano 2.5.3               File: hundred                                    </span>
 
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
                                <span style="background:white;color:black;">[ Read 100 lines ]</span>
 <span style="background:white;color:black;">^G</span> Get Help  <span style="background:white;color:black;">^O</span> Write Out <span style="background:white;color:black;">^W</span> Where Is  <span style="background:white;color:black;">^K</span> Cut Text  <span style="background:white;color:black;">^J</span> Justify   <span style="background:white;color:black;">^C</span> Cur Pos
 <span style="background:white;color:black;">^X</span> Exit      <span style="background:white;color:black;">^R</span> Read File <span style="background:white;color:black;">^\</span> Replace   <span style="background:white;color:black;">^U</span> Uncut Text<span style="background:white;color:black;">^T</span> To Spell  <span style="background:white;color:black;">^_</span> Go To Line
 ```
 
 ---
 
 # `nano`
 
 `nano` is a small and simple editor which helpfully shows its commands at the bottom (reminder, `^` means `Ctrl`):
 
 ```
 <span style="background:white;color:black;">^G</span> Get Help  <span style="background:white;color:black;">^O</span> Write Out <span style="background:white;color:black;">^W</span> Where Is  <span style="background:white;color:black;">^K</span> Cut Text  <span style="background:white;color:black;">^J</span> Justify   <span style="background:white;color:black;">^C</span> Cur Pos
 <span style="background:white;color:black;">^X</span> Exit      <span style="background:white;color:black;">^R</span> Read File <span style="background:white;color:black;">^\</span> Replace   <span style="background:white;color:black;">^U</span> Uncut Text<span style="background:white;color:black;">^T</span> To Spell  <span style="background:white;color:black;">^_</span> Go To Line
 ```
 
 You can use arrow keys to move around, input text as normal from where the cursor is.
 
 Key commands:
 
 * `Ctrl` + `W` ".highlight[w]here is" for searching the file
 * `Ctrl` + `O` "write .highlight[o]ut" to save changes
 * `Ctrl` + `X` "e.highlight[x]it" to get back to the shell
 
 ---
 
 # Try `nano`
 
 .exercise[
 1. Open a new file, `ten`, with `nano`:
 
   ```
   user@remote:~/scits-training/numbers/ $ nano ten
   ```
 
 2. Add numbers from 1 to 10 to it, on separate lines
 
 3. Save and exit `nano`
 
 4. Verify what's in the file using `cat`
 ]
 
 ---
 
 # `vim`
 
 `vim` (or, technically, "Vi IMproved") is one of two "Swiss knife" editors that
 most Linux professionals prefer to use (the other one being `emacs`).
 
 `vim` is available almost everywhere, and with proper configuration can do
 very sophisticated things.
 
 With power comes complexity, but for basic editing one doesn't have to remember a lot.
 
 If you wish to (later) explore `vim`, you can go through its built-in tutorial:
 
 ```
 vimtutor
 ```
 
 ---
 
 .center[
   [![:scale 60%](assets/exiting_vim.png)](http://www.commitstrip.com/en/2017/05/29/trapped/)
 ]
 
 ---
 
 # Organizing files and folders
 
 To recap, you should now be able to:
 
 * Navigate the file tree (with `cd`)
 * List folder contents (with `ls`)
 * Read and write files (with `nano`)
 
 Our goal now is:
 
 * Make new folders
 * To move and copy files and folders around
 * Delete files and folders
 
 ---
 
 # Creating new folders
 
 To create new folders, use the `mkdir` command:
 
 ```
 user@remote:~/scits-training/numbers/ $ cd ..
 user@remote:~/scits-training/ $ ls
 <span style="color:#3030ff;font-weight:bold">animals</span>  <span style="color:#3030ff;font-weight:bold">numbers</span>
 user@remote:~/scits-training/ $ mkdir new-folder
 user@remote:~/scits-training/ $ ls
 <span style="color:#3030ff;font-weight:bold">animals</span>  <span style="color:#3030ff;font-weight:bold">new-folder</span>  <span style="color:#3030ff;font-weight:bold">numbers</span>
 ```
 
 .mnemonic[
   .highlight[`mkdir`] stands for .highlight[make dir]ectory
 ]
 
 .exercise[
 1. Create `new-folder` as shown above
 
 2. Create directory `subfolder` inside it
 
 3. Verify with `ls`
 ]
 
 ---
 
 # Creating new folders
 
 `mkdir` will fail if the folder already exists:
 
 ```
 user@remote:~/scits-training/ $ mkdir new-folder
 mkdir: cannot create directory ‘new-folder’: File exists
 ```
 
 Using it with `-p` means "create if needed", and also works with chains of directories:
 
 ```
 user@remote:~/scits-training/ $ mkdir -p new-folder/subfolder/subsubfolder
 user@remote:~/scits-training/ $ ls -R new-folder
 new-folder:
 <span style="color:#3030ff;font-weight:bold">subfolder</span>
 
 new-folder/subfolder:
 <span style="color:#3030ff;font-weight:bold">subsubfolder</span>
 
 new-folder/subfolder/subsubfolder:
 user@remote:~/scits-training/ $
 ```
 
 ---
 
 # Moving files
 
 Move operations can be broken down into two cases:
 
 1. Moving files and folders between folders:
 
   `folder1/something` → `folder2/something`
 
 2. Renaming files and folders: 
 
   `something` → `other`
 
   Technically, it's "moving" from old name to new.
 
 Both cases are served with the `mv` command.
 
 .mnemonic[
   .highlight[`mv`] stands for .highlight[move]
 ]
 
 ---
 
 # Preparing for exercises
 
 ```
 user@remote:~/scits-training/ $ cd moving
 user@remote:~/scits-training/moving $ ls
 <span style="color:#3030ff;font-weight:bold">source</span>  <span style="color:#3030ff;font-weight:bold">destination</span>
 user@remote:~/scits-training/moving $ ls source
 A1  A10  A11  A12  A2  A3  A4  A5  A6  A7  A8  A9  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 user@remote:~/scits-training/moving $ ls source/subfolder
 B1  B2  B3  B4  B5  B6  B7  B8  B9
 user@remote:~/scits-training/moving $ ls destination
 user@remote:~/scits-training/moving $
 ```
 
 ---
 
 # Moving files
 
 To move something to another folder: .highlight[`mv NAME DESTINATION`],
 as long as the .highlight[`DESTINATION` is a directory that exists].
 
 ```
 user@remote:~/scits-training/moving $ mv source/A2 destination
 ```
 
 You can specify multiple things to move at the same time, including folders:
 
 ```
 $ mv source/A3 source/subfolder destination
 ```
 
 Moves both `source/A3` and `source/subfolder` into `destination`.
 
 .exercise[
   Move `subfolder` back into `source`
 ]
 
 ---
 
 # Renaming
 
 Renaming is easy: `mv OLDNAME NEWNAME`, if `NEWNAME` is *not* a directory.
 
 For example, let's rename `destination` to `dest`:
 
 ```
 user@remote:~/scits-training/moving $ mv destination dest
 ```
 
 If you're renaming something in another folder, you must specify the path twice:
 
 ```
 $ mv source/A4 source/A40
 ```
 
 .exercise[
 1. Rename `dest` back into `destination`
 
 2. Rename `source/subfolder/B1` into `source/subfolder/B10`
 ]
 
 ---
 
 # Move + rename
 
 .exercise[
 Try the following:
 
 ```
 user@remote:~/scits-training/moving $ mv source/A5 A50
 ```
 
 Use `ls` to understand what happened (`-R` may help)
 ]
 
 --
 
 **Answer:** Since there is no path for the second name, it moved into the current directory
 and got renamed:
 
 `~/scits-training/moving/source/A5`
 
 ↓
 
 `~/scits-training/moving/A50`
 
 ---
 
 # Copying
 
 Copying is done with `cp`
 
 .mnemonic[
   .highlight[`cp`] stands for .highlight[copy].
 ]
 
 Syntax is the same:
 
 * For copying to another directory, `cp NAME DESTINATION`
 * For copying to another name, `cp OLDNAME NEWNAME`
 
 .exercise[
 1. Copy `source/A6` and `source/A7` into `destination`
 
 2. Copy `source/A6` into `source/A66`
 ]
 
 ---
 
 # Copying folders
 
 `cp`, unlike `mv`, will not copy directories by default:
 
 ```
 $ cp source/subfolder destination
 cp: omitting directory 'source/subfolder'
 ```
 
 You need to use `-R` to copy folders together with their content
 
 ```
 $ cp -R source/subfolder destination
 ```
 
 .mnemonic[
 .highlight[`-R`] stands for .highlight[r]ecursive
 ]
 
 ---
 
 # Deleting
 
 To remove files or folders, use `rm`
 
 .mnemonic[
   .highlight[`rm`] stands for .highlight[rem]ove
 ]
 
 * `rm NAME` to remove a file
 * `rm -r FOLDER` to remove a folder
 
 You can pass several names at once:
 
 ```
 $ rm destination/subfolder/B10 destination/subfolder/B2
 ```
 
 ---
 
 # `rm` is unrecoverable!
 
 When you delete files and folders with `rm`, you should be aware that there is no concept of "Trash".
 
 Anything you delete (or overwrite) is lost with no easy way to recover.
 
 You can use a flag `-i` to ask before any destructive operation.
 
 ```
 user@remote:~/scits-training/moving $ cp -i -R source/subfolder destination
 cp: overwrite 'destination/subfolder/B1'?
 ```
 
 On the other hand, sometimes you want to override those confirmations, especially for `rm` – you can do it with `-f`.
 
 .mnemonic[
   .highlight[`-i`] stands for .highlight[interactive]  
   .highlight[`-f`] stands for .highlight[force]
 ]
 
 ---
 
 # Wildcards
 
 There are many A-files in `source`:
 
 ```
 user@remote:~/scits-training/moving $ ls source
 A1  A10  A11  A12  A40  A6  A66  A7  A8  A9  <span style="color:#3030ff;font-weight:bold">subfolder</span>
 ```
 
 We may want to copy them all at once. We can use wildcards:
 
 * .highlight[`*`] in a name means "any amount of any characters"
   * For example, .highlight[`A*`] can mean `A`, `A1` and `A10`
 * .highlight[`?`] in a name means "any single character"
   * For example, .highlight[`A?`] can mean `A1`, `A6` but not `A10`
 
 The wildcards will .highlight[not jump through directories]:
 
 * .highlight[`*1`] can mean `A1`, `A11`, but not `subfolder/B1`
 * .highlight[`\*/\*`] can match `subfolder/B1`
 
 ---
 
 # Wildcard quiz (1/3)
 
 Which of the following names match the pattern .highlight[`A\*a\*`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>AAa</code></td>
     </tr>
     <tr>
       <td class="N"><code>A/a</code></td>
     </tr>
     <tr>
       <td class="N"><code>aaA</code></td>
     </tr>
     <tr>
       <td class="N"><code>CBAcba</code></td>
     </tr>
     <tr>
       <td class="N"><code>abcABC</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Wildcard quiz (2/3)
 
 Which of the following names match the pattern .highlight[`A?a?`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="N"><code>AAa</code></td>
     </tr>
     <tr>
       <td class="Y"><code>AAaa</code></td>
     </tr>
     <tr>
       <td class="Y"><code>Aaaa</code></td>
     </tr>
     <tr>
       <td class="N"><code>AAaaa</code></td>
     </tr>
     <tr>
       <td class="N"><code>aAAaa</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Wildcard quiz (3/3)
 
 Which of the following patterns match the name .highlight[`A110`]
 
 <div class="center">
   <table class="quiz">
     <tr>
       <td class="Y"><code>A*</code></td>
     </tr>
     <tr>
       <td class="Y"><code>*</code></td>
     </tr>
     <tr>
       <td class="Y"><code>*A*</code></td>
     </tr>
     <tr>
       <td class="N"><code>*A</code></td>
     </tr>
     <tr>
       <td class="Y"><code>A???</code></td>
     </tr>  
   </table>
 </div>
 
 ---
 
 # Using wildcards
 
 Putting a name with a wildcard is equivalent to putting several names:
 
 ```
 $ cp source/A6* destination
 ```
 
 is equivalent to
 
 ```
 $ cp source/A6 source/A66 destination
 ```
 
 So, you can use wildcards in any command that expects multiple files.
 
 ---
 
 # Try wildcards
 
 Use wildcards to do the following, from `~/scits-training/moving`:
 
 .exercise[
 1. List all files starting with `A` inside `source/` (use `ls` with a pattern).
 
 2. Copy all files starting with `B` from `source/subfolder` into `destination`.
 
 3. Move all files starting with `A1` from `source` into `destination`.
 
 4. Delete all files starting with `A` from `destination`.
 ]
 
 ---
 
 # Moving data in and out
 
 So far we have moved the data around on the system itself.
 
 It doesn't help if you want to load external data or
 download the results of your programs.
 
 --
 
 Perhaps, it's your own system and you have access to cloud storage
 or external storage devices.
 
 Sometimes, you have a shared network folder between your computer
 and the target system.
 
 But how to do it, if your only interface to the server is SSH?
 
 --
 
 We will cover two ways:
 
 1. Downloading data from the Internet with `wget`
 
 2. Copying data between computers with `scp`
 
 ---
 
 # Downloading from the shell
 
 Sometimes, the data you need is a file on the Internet.
 
 `wget` is the simplest-to-use tool for it:
 
 ```
 user@remote:~/scits-training/moving $ cd ..
 user@remote:~/scits-training/ $ wget https://example.com/
 --2017-09-12 12:00:00--  https://example.com/
 Resolving example.com (example.com)... 93.184.216.34
 Connecting to example.com (example.com)|93.184.216.34|:443... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 1270 (1.2K) [text/html]
 Saving to: ‘index.html’
 
 100%[======================================>] 1,270       --.-K/s   in 0s
 
 2017-09-12 12:00:00 (36.2 MB/s) - ‘index.html’ saved [1270/1270]
 
 user@remote:~/scits-training/ $ tail index.html
 ```
 
 .mnemonic[
   .highlight[`wget`] stands for .highlight[Web get]
 ]
 
 ---
 
 # Downloading from the shell
 
 Notable option: renaming the file immediately.
 
 * .highlight[-O] (for .highlight[o]utput) chooses a specific file to write to
 
 ```
 $ wget https://tools.ietf.org/rfc/rfc1149.txt -O april.txt
 [...]
 $ less april.txt
 ```
 
 As usual, use `man wget` to see more options.
 
 It can work with HTTP/HTTPS/FTP-hosted files.
 
 ---
 
 # Transferring files between systems
 
 To send files between two computers using SSH, the simplest command is `scp`.
 
 .mnemonic[
   .highlight[`scp`] stands for .highlight[secure copy]
 ]
 
 `scp` behaves a lot like `cp`, but you can provide locations on other computers.
 
 How to use `scp` on your own machine depends on the OS.
 
 ---
 
 # `scp` on Windows
 
 While PuTTY includes a command-line client `pscp` with the same functions,
 it may be better to use a GUI client `WinSCP`.
 
 It can be downloaded from https://winscp.net/
 
 You can then connect using SCP (or SFTP) with your normal credentials
 and transfer files between your PC and the remote:
 
 ![](assets/winscp.png)
 
 ---
 
 # `scp` on Linux / MacOS
 
 From your .highlight[local] terminal, you can transfer a file from UBELIX:
 
 ```
 user@local:~ $ scp user@submit.unibe.ch:~/scits-training/numbers/hundred .
 [..some authentication..]
 user@local:~ $ less hundred
 ```
 
 `scp`'s parameters work similarly to `cp`, but you can refer to files on other systems
 by adding `user@remote:` to the path.
 
 It works both ways, and can rename as well:
 ```
 $ scp hundred user@submit.unibe.ch:~/scits-training/numbers/another_hundred
 ```
 
 ---
 
 # Moving data in and out
 
 .exercise[
 1. Copy all of the `B`-files from `moving/source/subfolder` to your computer with one command (use wildcards).
 
 2. Copy some folder from your computer to the home folder of the remote system (use `-r`).
 ]
 
 --
 
 In addition to `scp`, there's a command that works better
 for repeatedly copying large folders with small changes: .highlight[`rsync`].
 
 It will not be covered here, but look up information on it if it's your use case.
 
 ---
 
 name: part2
 class: center, middle
 
 # Introduction to Linux
 
 ## Part II
 
 `https://goo.gl/Vg3iXW#part2`
 
 ---
 
 # Agenda
 
 1. Linux resources you can use
 2. Standard input/output and its redirection
 3. UNIX pipelines
 4. Background processes 
 5. Durable sessions with `screen`
 6. File ownership and permissions
 7. Shell scripting basics
 8. Environment and customization
 9. (Extra credits)
 
 ---
 
 # What Linux resources can I use?
 
 To do development and run light workloads:
 
 * Your own computer may already run Linux.
 
 * You can install Linux in a virtual machine.
 
   I recommend Virtualbox for personal use.
 
 * If you're running Windows 10, you can install  
   [Windows Subsystem for Linux](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux)
 
 ---
 
 # What if it's not enough?
 
 To create persistent services:
 
 * Ask your group's sysadmin for servers/VM resources.
 
 * UniBe Informatikdienste offers virtual machines.
 
 * Cloud resources: [SWITCHengines](https://www.switch.ch/engines/), other cloud services.
 
 To run heavy calculations:
 
 * [UBELIX Linux cluster](https://docs.id.unibe.ch/ubelix/ubelix-101).
 
 * Your group may have in-house infrastructure.
 
 * Again, cloud services.
 
 ---
 
 # Prepare for the tutorial
 
 You should be either running Linux, or connected to a Linux system.
 
 Log in to UBELIX, or inform me that you need access to another system.
 
 Execute the following to set up training (if you haven't already):
 ```
 $ wget https://scits.math.unibe.ch/script -O - | /bin/bash
 ```
 
 ---
 
 # Processes and their input/output
 
 When you're connected to a Linux system,
   what you normally see is the shell prompt, awaiting input:
 
 ```
 user@host:~ $
 ```
 
 A .highlight[process] called shell is responsible for input/output at this moment.
 
 By default, the input is either a keyboard connected to the system,
   or your keypresses being relayed over the network.
 
 The output, by default, is the screen connected to the system,
   or text being relayed for display over the network.
 
 ---
 
 # Processes and their input/output
 
 Those are called .highlight[standard input] and .highlight[standard output],
   or .highlight[STDIN/STDOUT].
 
 .center[
   ![](assets/shell_stdio.svg)
 ]
 
 --
 
 If the user issues a command that calls another program, the shell
   creates a .highlight[child process] and attaches the input/output to it.
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 The shell will wait until the program terminates,
   after which STDIN/STDOUT get reattached and a prompt is displayed.
 
 ---
 
 # Redirecting output
 
 Sometimes, we want to capture what a command is outputting to the screen.
 
 For example, suppose that we want to save into a file the directory structure returned by `ls -R`:
 
 ```
 user@host:~ $ ls -R ~/scits-training
 scits-training:
 animals  moving  numbers  io  scripts
 [...]
 user@host:~ $
 ```
 
 We can instruct the shell to .highlight[redirect] the standard output:
 
 .center[
   ![](assets/stdout_redirect.svg)
 ]
 
 ---
 
 # Redirecting output
 
 To redirect the output, we add .highlight[`> FILE`] to the command:
 
 ```
 user@host:~ $ cd ~/scits-training/io
 user@host:~/scits-training/io $ ls -R ~/scits-training > listing
 user@host:~/scits-training/io $ cat listing
 scits-training:
 animals  moving  numbers  io  scripts
 [...]
 ```
 
 This will overwrite the contents of `FILE` (if any) with the output of the command.
 
 The file will be created, if it does not exist yet.
 
 ---
 
 # Appending output
 
 Sometimes we don't want to overwrite (sometimes called "clobber") the file
   with new contents and add them to the end instead.
 
 To do that, use .highlight[`>> FILE`] instead of `> FILE`.
 
 .exercise[
 1. Try running the command `date` to see what it outputs.
 
 2. Run `date` 3 times, appending the output to a file `date.log`.
 
 3. Verify with `cat` that the file contains 3 records.
 ]
 
 ---
 
 # Error output
 
 Try saving the output of a command with errors
 and you'll see that it still outputs to the screen:
 
 ```
 user@host:~/scits-training/io $ ls , > listing
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $
 ```
 
 --
 
 This is intentional: Linux actually has two output streams for
   its command line, STDOUT for normal data and .highlight[STDERR] for errors.
 
 .center[
   ![](assets/stderr.svg)
 ]
 
 This simplifies debugging: errors are separate from data.
 
 ---
 
 # Error output
 
 STDERR is not redirected when using `>` or `>>`:
 
 .center[
   ![](assets/stderr_noredirect.svg)
 ]
 
 --
 
 It's possible to redirect it as well with .highlight[`2>`] or .highlight[`2>>`]:
 
 ```
 user@host:~/scits-training/io $ ls .. , > listing 2> errors
 user@host:~/scits-training/io $ cat errors
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $ cat listing
 ..:
 animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Discarding output
 
 Sometimes we don't need output at all.
 
 In this case, we can redirect it to a special file, .highlight[`/dev/null`]
 
 It's a .highlight[device] that will accept any input, discarding it immediately.
 
 --
 
 For example, one might want to silence errors:
 
 ```
 user@host:~/scits-training/io $ ls .. , 2> /dev/null
 ..:
 animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Interactive input
 
 Most commands we've seen don't require any interactive input.
 
 `tr` (for .highlight[translate]) is a command that transforms its input:
   it substitutes some characters with others.
 
 For example, `tr 'a-z' 'A-Z'` would translate all lowercase letters into uppercase.
 
 --
 
 Let's try this:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 ```
 
 ---
 
 # Interactive input
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 But enter doesn't stop it!
 BUT ENTER DOESN'T STOP IT!
 ```
 
 A problem: text can contain many lines, and the program won't know when to stop.
 
 --
 
 We can terminate the program with Ctrl+C, but it actually expects an .highlight[end of input].
 
 We can signal end of input with Ctrl+D on an empty line (or pressing it twice).
 
 --
 
 .exercise[
   What happens if we press Ctrl+D while back at the shell prompt?
 ]
 
 ---
 
 # Redirecting input
 
 What if we want to use a file as an input in a command that doesn't accept files as arguments,
   we need instruct the shell to use the file as the program's standard input:
 
 .center[
   ![](assets/stdin_redirect.svg)
 ]
 
 This is done with adding .highlight[`< FILE`] to the command.
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' < errors
 LS: CANNOT ACCESS ,: NO SUCH FILE OR DIRECTORY
 ```
 
 ---
 
 # Redirects
 
 .exercise[
 1. Combine input and output redirection to save the output of last `tr` command into `errors.uppercase`
 
 2. Use `cat` to verify the saved output.
 ]
 
 ---
 
 # Pipelines
 
 We have shown how to save outputs to a file, and further process files as inputs.
 
 Sometimes, we don't need to save this intermediate representation.
 In that case, we can directly connect the output of one program to the input of another
 with .highlight[pipes].
 
 To do so, separate two commands with .highlight[`|`]:
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 ---
 
 # Pipelines
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 
 Given this command, shell starts two processes in parallel and ties their respective output and input together.
 
 Standard input/output is connected at the ends of the chain:
 
 .center[
   ![](assets/pipe.svg)
 ]
 
 --
 
 Such .highlight[pipelines] can be longer than two commands, and can be combined with file redirects.
 
 ---
 
 # Filtering output
 
 One extremely useful command used in pipes is `grep`.
 
 It allows to search for text patterns. Example:
 
 ```
 user@host:~/scits-training/io $ ls .
 date.log
 errors
 errors.uppercase
 listing
 user@host:~/scits-training/io $ ls . | grep log
 date.log
 ```
 
 `grep` is versatile:
 
 * Can be used with .highlight[regular expression] patterns
 * Can search for non-matching lines (with `-v`)
 * Can search in files
 * Can print where the match happened in a file
 
 See `man grep` or Google for more examples.
 
 ---
 
 # Pipelines and errors
 
 You will notice that all errors are still output normally:
 
 ```
 user@host:~/scits-training/io $ ls . , | tr 'a-z' 'A-Z'
 ls: cannot access ,: No such file or directory
 .:
 DATE.LOG
 ERRORS
 ERRORS.UPPERCASE
 LISTING
 ```
 
 --
 
 As before, errors are not normally redirected, and collected from all processes in the pipe:
 
 .center[
   ![](assets/pipe_err.svg)
 ]
 
 ---
 
 # Background jobs
 
 Recall that, when running a command, the shell waits until it is terminated: all input goes to the program (or nowhere).
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 Sometimes, we don't need to wait until the program terminates – we actually want it running in background.
 
 ---
 
 # Background jobs
 
 If you specify .highlight[`&`] at the end of the command, the shell will start it, but keep control of STDIN:
 
 .center[
   ![](assets/job_stdio.svg)
 ]
 
 Instead of a .highlight[foreground] process, it becomes a .highlight[background] job.
 
 --
 
 You are immediately returned to the shell and can run other commands while the job executes.
 
 Note that both the shell and the background job are connected to STDOUT.
 Redirect output to prevent mix-ups.
 
 ---
 
 # Background jobs
 
 Compare:
 
 ```
 user@host:~/scits-training/io $ sleep 3
 user@host:~/scits-training/io $ sleep 3 &
 [1] 12231
 user@host:~/scits-training/io $
 ```
 
 Here, `[1]` is the .highlight[job number], and `12231` is the .highlight[process ID], or PID.
 
 After 3 seconds and when another command finishes (you can just press Enter for an empty command),
 you'll be informed that the job terminated:
 
 ```
 user@host:~/scits-training/io $
 [1]+  Done                    sleep 3
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Listing jobs
 
 You can list running background jobs with .highlight[`jobs`]:
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12232
 user@host:~/scits-training/io $ sleep 0 &
 [2] 12233
 user@host:~/scits-training/io $ jobs
 [1]-  Running                 sleep 100 &
 [2]+  Done                    sleep 0
 ```
 
 ---
 
 # Terminating jobs
 
 You can forcibly .highlight[terminate] a job with the .highlight[`kill`] command,
   which accepts either PID or job ID (with `%`):
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12234
 user@host:~/scits-training/io $ kill 12234
 user@host:~/scits-training/io $ jobs
 [1]+  Terminated              sleep 100
 
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12235
 user@host:~/scits-training/io $ kill %1
 ```
 
 You can search for more process IDs to terminate with .highlight[`ps ax`], in case something is misbehaving.
 
 ---
 
 # Stopped jobs
 
 Background jobs have nothing connected to their standard input.
 
 If a background job cannot continue without user input, it will .highlight[stop],
 which the shell will signal to you:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' &
 [1] 12236
 user@host:~/scits-training/io $
 [1]+  Stopped                 tr /a-z/ /A-Z/
 user@host:~/scits-training/io $
 ```
 
 --
 
 You can bring a job to .highlight[foreground] to pass STDIN from the shell
 to the running job with `fg` (or `fg %N` for a specific job number):
 
 ```
 user@host:~/scits-training/io $ fg
 tr /a-z/ /A-Z/
 You are now talking to the job
 YOU ARE NOW TALKING TO THE JOB
 ```
 
 ---
 
 # Stopping and resuming programs
 
 You can stop most currently-running programs with `Ctrl`+`Z`:
 
 ```
 user@host:~/scits-training/io $ sleep 100
 ^Z
 [1]+  Stopped                 sleep 100
 user@host:~/scits-training/io $
 ```
 
 --
 
 From there, you can use `fg` to resume normal execution of the program,
   or use .highlight[`bg`] to let it continue to run in the background.
 
 ```
 user@host:~/scits-training/io $ bg
 [1]+ sleep 100 &
 user@host:~/scits-training/io $ jobs
 [1]+  Running                 sleep 100 &
 ```
 
 ---
 
 # Background jobs are fragile
 
 What will happen if you start a background job, and then close the terminal?
 
 --
 
 Closing the terminal (or disconnecting the SSH session) kills the shell you were talking to.
 Since the job was a .highlight[child process] of that shell, it will also be killed.
 
 --
 
 A minor inconvenience if you're working on your own machine (you can just leave the terminal open),
 but a much bigger problem with remote connections.
 
 If the connection is broken, the shell is also terminated along with all processes launched from it.
 
 How to protect against it?
 
 ---
 
 # `screen`
 
 To protect your session, you can use .highlight[`screen`].
 
 `screen` starts a new shell that exists independently of your current one.
 
 Even if the current shell dies (e.g. because you disconnected),
 the shell running in `screen` will continue together with all its child processes.
 
 --
 
 Starting a new `screen` session is simple:
 
 ```
 user@host:~/scits-training/io $ screen
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Reattaching to `screen`
 
 Now suppose your connection was terminated.
 
 Close the terminal where it is running to simulate that, then log in again.
 
 --
 
 You can use .highlight[`screen -ls`] to list active sessions:
 
 ```
 user@host:~ $ screen -ls
 There is a screen on:
         13383.pts-2.host        (11/09/17 03:02:23)     (Detached)
 1 Socket in /var/run/screen/S-user.
 ```
 
 --
 
 You can attach to a screen session (possibly detaching it first, if it's being used somewhere)
   with .highlight[`-dR`] (for .highlight[detach, reattach])
 
 ```
 user@host:~ $ screen -dR
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Controlling `screen`
 
 `screen` can be used for other things, such as having multiple parallel shell sessions open.
 
 Controlling screen consists of pressing .highlight[`Ctrl`+`A`], then a screen-specific command.
 
 For example,
 
 * .highlight[`c`] will .highlight[create] a new shell within screen
 * .highlight[`n`] will switch to the .highlight[next] shell
 * .highlight[`d`] will .highlight[detach] from `screen`, returning you to the original shell
 
 Finally, you can use .highlight[`?`] to access built-in help,
 or use `man screen` for a more detailed manual.
 
 --
 
 Another popular alternative to `screen` is `tmux`. It will not be covered by this tutorial,
   but is worth looking into.
 
 ---
 
 # Users and groups
 
 Before we discuss permissions, we need to understand users and groups in Linux.
 
 A .highlight[user] is a unit of access control; it has a set of credentials to access the system
   and .highlight[owns] some files on it.
 
 A .highlight[group] is a collection of users to facilitate shared access to resources.
   A user can belong to many groups but one group is considered primary.
 
 You can use `id` to check your user and groups:
 
 ```
 akashev@submit01:~ $ id
 uid=7265(akashev) gid=1109(math) groups=1109(math),902(l_gaussian)
 ```
 
 Here, `akashev` is my user, `math` is my primary group and `l_gaussian` is another group I belong to.
 
 ---
 
 # Permissions: `rwx`
 
 Each file and directory in UNIX filesystems has 3 permissions (for a particular user).
 
 Regular files:
 * .highlight[`r`], or .highlight[Read], means that you can read the contents of a file.
 * .highlight[`w`], or .highlight[Write], means that you can modify the file.
 * .highlight[`x`], or .highlight[eXecute], means that the file may be launched as a program.
 
 Directories:
 * .highlight[`r`] means that you can read the list of files within the directory.
 * .highlight[`w`] means that you can add or delete files from the directory.
 * .highlight[`x`] means you can .highlight[traverse] the folder: enter it with `cd` and read the contents of its files.
 
 ---
 
 # Inspecting permissions
 
 Try running `ls -la` to see permissions on files and folders:
 
 ```
 $ ls -la
 total 20
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 drwxrwxr-x 6 user group 4096 Sep 10 23:06 ..
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 -rw-rw-r-- 1 user group   47 Sep 11 00:50 errors
 -rw-rw-r-- 1 user group   30 Sep 11 01:09 listing
 ```
 
 We're interested in the first column: the cryptic `drwxrwxr-x` and `-rw-rwr--`,
 which are called .highlight[mode].
 
 --
 
 * The first character denotes the .highlight[file type].
   * `-` means "regular file".
   * `d` means "directory".
 * The rest is divided in groups of three:
   * Access for the owner
   * Access for the group
   * Access for everyone else
 
 ---
 
 # File ownership
 
 ```
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 Each file in a UNIX filesystem has an .highlight[owner] and a .highlight[group] attached.
 
 In the example above, `user` is the owner and `group` is the designated group.
 
 Note that the user .highlight[doesn't have to be in the assigned group].
 
 ---
 
 # Effective permissions
 
 ```
 -rwxr-x--- 1 user group   90 Sep 10 23:08 script
 ```
 
 To determine which permissions apply, the following is checked:
 
 * If the user is the owner, the first set applies (`rwx`, full permissions)
 * If the user is in the designated group, the second set applies (`r-x`, so cannot write)
 * For all other users, the third set applies (`---`, so cannot do anything)
 
 --
 
 A special user, .highlight[superuser] (normally called .highlight[root]),
   can completely disregard permissions and do anything to any file on the system.
 
 ---
 
 # Permissions: first match applies
 
 Note that the system does not apply "best" permissions – only the first set that matches.
 
 Let's reverse the situation:
 
 ```
 ----r-xrwx 1 user group   90 Sep 10 23:08 script
 ```
 
 For this file, the owner cannot do anything to the file, anyone in `group` cannot modify it,
   but everyone else has full permissions.
 
 Note: the owner can always change a file's permissions.
 
 ---
 
 # Modifying permissions
 
 To modify a file's permissions, use .highlight[`chmod CHANGES FILE`]
 
 .mnemonic[
   `chmod` stands for .highlight[change mode].
 ]
 
 Possible changes:
 * .highlight[`+r`], .highlight[`+w`], .highlight[`+x` add] permissions.
   Can combine: `+rw`
 * .highlight[`-r` removes] permissions.
 * .highlight[`=r`] sets pemissions to .highlight[exactly] `r--`.
 * Prefix .highlight[`u`] changes permissions for the .highlight[user], e.g. `u+r`.
 * Prefix .highlight[`g`] changes permissions for the .highlight[group], e.g. `g+rw`.
 * Prefix .highlight[`o`] changes permissions for .highlight[others], e.g. `o-w`.
 * Prefix .highlight[`a`] or no prefix changes permissions for .highlight[all] three sets.
 * An .highlight[octal number] (e.g. 750) sets permissions to a specific configuration
 (in this case, `rwxr-x---`).
 
 ---
 
 # Modifying permissions
 
 Several changes can be applied at once, separated by commas:
 
 ```
 user@host:~/scits-training/io $ ls -la date.log
 -rwxrw-r-- 1 user group   90 Sep 10 23:08 date.log
 
 user@host:~/scits-training/io $ chmod u+x,g=rx,o-r date.log
 
 user@host:~/scits-training/io $ ls -la date.log
 -rwxr-x--- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 .exercise[
 Modify permissions on the file to be `r-xr--rw-`
 ]
 
 ---
 
 # Changing ownership
 
 Similarly to `chmod`, the .highlight[`chown`] command allows changing a file's owner and group.
 
 * `chown USER FILE` changes the owner
 * `chown :GROUP FILE` changes the group
 * `chown USER:GROUP FILE` changes both
 
 Note: once the owner is changed, the old owner no longer can modify access to the file.
 
 .highlight[For this reason, only administrators can change the file owner, or assign a group the owner is not part of.]
 
 .exercise[
 Use `groups` to list groups you belong to.
 
 Change a file's group to one of them, and then back to the original one.
 ]
 
 ---
 
 # Shell scripting
 
 Shell is not just an interface to launch other programs;
 it comes with its own scripting language to automate complex tasks.
 
 You can have variables, loops, conditionals – a full-featured programming language.
 
 We will only show the very basics.
 
 .exercise[
   Navigate to `~/scits-training/scripts` and open `boom.sh` in your favourite editor (nano, vim)
 ]
 
 ---
 
 layout: true
 
 # Shell scripting
 
 ```bash
 #!/bin/bash
 
 # I hope you get the reference
 echo "Someone set up us the bomb."
 for i in {5..1}
 do
   echo "$i.."
   sleep 1
 done
 explosion="Boom!"
 echo $explosion
 ```
 
 ---
 
 The first line of the script is special:
 ```bash
 #!/bin/bash
 ```
 
 It's called a .highlight["shebang"] (for .highlight[shell] and "!" .highlight[bang]).
 
 It tells the shell what to execute the rest of the script with.
 Since we're writing a `bash` shell script, we put there the path to `/bin/bash` itself.
 
 ---
 
 Other lines starting with .highlight[`#`] are .highlight[comments]
 ```bash
 # I hope you get the reference
 ```
 
 They are ignored by `bash` and are used to leave notes to yourself or others.
 
 ---
 
 .highlight[`echo`] command outputs its arguments to STDIN.
 
 ```bash
 echo "Someone set up us the bomb."
 ```
 
 Quotes are used to make text with spaces in it a single argument; here, they are optional.
 
 ---
 
 .highlight[`for`] designates a .highlight[loop]: a .highlight[variable `i`] will change from 5 to 1.
 
 ```bash
 for i in {5..1}
 do
   # something
 done
 ```
 
 The code in `# something` will repeat with `i` as 5, 4, 3, 2 and 1.
 
 `do` and `done` delimit the bounds of the loop.
 
 ---
 
 One can use the variable in expressions .highlight[prefixed by `$`], i.e. .highlight[`$i`]:
 
 ```bash
 echo "$i.."
 ```
 
 If there is ambiguity as to where a variable name ends, use braces: .highlight[`${i}`],
   e.g. `"Sample ${i}A"` for "Sample 1A", etc.
 
 ---
 
 Variables can also be simply .highlight[assigned to]:
 
 ```bash
 explosion="Boom!"
 echo $explosion
 ```
 
 The lack of spaces around `=` is .highlight[significant].  
 Otherwise Bash will try to execute `explosion` as a command.
 
 ---
 
 layout: false
 
 # Running a script
 
 OK, suppose we wrote the above script. How to execute it?
 
 1. We need to make sure that it's allowed to execute:
 
   ```
   user@host:~/scits-training/scripts $ chmod +x boom.sh
   ```
 
 2. For security reasons, the current directory is not automatically considered when
   starting other programs. We need to explicitly refer to it:
 
   ```
   user@host:~/scits-training/scripts $ ./boom.sh
   ```
 
 .exercise[
 1. Execute the script, saving its output to a file.
 
 2. Modify the script to count down from 10.
 ]
 
 ---
 
 # Scripting, take two
 
 The next script you will type out yourselves.
 
 Open `beer.sh` in your favourite editor.
 
 We'll write a simple script to determine if a user is old enough to drink beer.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 ```
 
 Any bash script should start with an appropriate shebang.
 
 We want to ask the user for his/her age; we can use the .highlight[`read`] command.
 
 ```bash
 # -n prevents a line break, and note the extra space
 echo -n "What's your age? "
 read age
 ```
 
 This will display a promt for the user and wait for input.
 The result is then stored in the variable `$age`.
 
 For simplicity, we will not check that the input is indeed a valid number.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 ```
 
 We need to make a decision based on age; we need an if-then-else construct.
 
 ```bash
 if [ $age -lt 16 ]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .highlight[`fi`] here is `if` reversed, to close the .highlight[`if`] statement.
 
 Conditionals in bash are a bit clunky, but .highlight[`-lt`] here stands for .highlight[less than].
   Again, the whitespace here is .highlight[significant].
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 if [ $age -lt 16 ]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .exercise[
 1. Save this script to `beer.sh`.
 
 2. Change the file's mode to allow execution.
 
 3. Test the script with different values.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's add a little personal touch.
 
 `whoami` is a command that returns the username. Let's edit `beer.sh` to use it:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
 else
   echo "$(whoami), you're old enough, have a beer!"
 fi
 ```
 
 .highlight[`$(something)`] allows you to execute a command and substitute the result within another command.
 
 .exercise[
 Test the new additions.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's read the age from the command line arguments.
 
 `bash` automatically populates .highlight[`$0`] with the .highlight[name of the executable],
 and .highlight[`$1`], .highlight[`$2`] and so on with .highlight[arguments].
 
 Let's use `$1` as age if it's defined:
 
 ```bash
 if [ $1 ]
 then
   age=$1
 else
   echo -n "What's your age? "
   read age
 fi
 ```
 
 .exercise[
 Test that `./beer.sh` now automatically gets the age from its first argument, and still asks if no argument is provided.
 ]
 
 ---
 
 # Return values
 
 Whenever a program terminates, it returns a single integer to the shell that called it;
   it's called the .highlight[return value].
 
 By convention:
 * .highlight[0] means "no error".
 * any .highlight[non-zero value] means "some kind of error".
 
 Let's return appropriate values:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
   exit 1
 else
   echo "$(whoami), you're old enough, have a beer!"
   exit 0
 fi
 ```
 
 ---
 
 # Chaining commands
 
 You can chain commands in shell with .highlight[`;`] or .highlight[`&&`].
 
 .highlight[`;`] will execute commands one by one, regardless of errors.
 
 ```
 $ command1; command2
 ```
 
 .highlight[`&&`] will only execute the next command only if the previous one returned 0, i.e. finished without errors.
 ```
 $ command1 && command2
 ```
 
 .exercise[
 1. Apply the return value changes to `beer.sh`
 
 2. Test it with `./beer.sh && echo 'Cheers!'`
 ]
 
 ---
 
 # Environment variables
 
 Your profile files can set various .highlight[environment variables]:
 snippets of data inherited by programs running from shell.
 
 You can see your current environment variables with:
 
 ```
 $ env | less
 ```
 
 --
 
 Some programs rely on environmental variables to change their behavior. Example:
 
 ```bash
 # Will replace the default editor with vim in some commands
 export EDITOR=vim
 ```
 
 ---
 
 # Environment variables
 
 Environment variables work similarly to variables in a script, except for the extra
   command .highlight[export], which propagates this variable to child processes.
 
 By convention, environment variables are UPPERCASE.
 
 --
 
 .exercise[
 1. Set an ordinary varable A (`A='Hello'`) and environment variable B (`export B='World'`).
 
 2. Use `echo $A` and `echo $B` to display them.
 
 3. Add `echo $A` and `echo $B` to a script and execute it. What changes?
 ]
 
 ---
 
 # `$PATH` variable
 
 An important variable is `$PATH`.
 
 It's a colon-separated list of directories which are searched when you try to run a program by name.
 
 Notably, the current directory is not in `$PATH`.
 
 If you have created some own scripts/programs and want them to be available by name from anywhere,
   you can put them in a folder (e.g. `~/bin`) and add it to `$PATH`:
 
 ```bash
 export PATH="$PATH:$HOME/bin"
 ```
 
 ---
 
 # `$PATH` variable, example
 
 .exercise[
 1. Try running `beer.sh` directly y name. It fails.
 
 2. Add the folder that holds it to `$PATH`:
 
 ```bash
 export PATH="$PATH:~/scits-training/scripts"
 ```
 
 3. Try running `beer.sh` now.
 
 4. Try going somewhere else (`cd ~`) and run it.
 ]
 
 ---
 
 # Aliases
 
 If you use a certain command often, you can define a short name for it.
 
 For example, if you want a shorter name for `ls -lh` because you always want to see human-readable sizes,
 you can make an alias:
 
 ```
 $ alias lh="ls -lh"
 $ lh
 total 26M
 -rw-r--r-- 1 user group 25M Sep 11 07:22 big_file
 -rw-r--r-- 1 user group 735 Sep 11 07:22 description
 -rw-r--r-- 1 user group   0 Sep 11 07:22 empty_file
 -rw-r--r-- 1 user group 551 Sep 11 07:22 naming
 drwxr-xr-x 0 user group 512 Sep 11 07:22 subfolder
 ```
 
 ---
 
 # Making customizations permanent
 
 To make above tweaks permanent, they need to be added either to `.bash_profile` or `.bashrc`.
 
 Then they will apply on each opened shell.
 
 * `.bash_profile` is .highlight[sourced] at most once. Put things there that shouldn't be called multiple times.
 
 * `.bashrc` is sourced almost every time `bash` is called, except for initial SSH shell.
   To be safe, you can "include" `.bashrc` into `.bash_profile` like this:
 
 ```bash
 # In .bash_profile
 # -f tests that file exists
 # source executes commands in the current shell
 if [ -f ~/.bashrc ]; then
   source ~/.bashrc
 fi
 ```
 
 ---
 
 class: center, middle
 
 # Extra credits
 
 ---
 
 # Searching through history
 
 There's a way to quickly search through previous commands.
 
 `Ctrl`+`R` opens "reverse search" mode. Enter some pattern and the closest command in history that matches will be shown.
 
 To look into older commands, press `Ctrl`+`R` again, or `Ctrl`+`C` to abort.
 
 ---
 
 # Custom shell prompt
 
 The variable `$PS1` contains the format template for your shell prompt.
 
 Throughout this training, you saw the following prompt:
 
 ```
 user@host:~ $ 
 ```
 
 You can customize it! For example:
 ```
 user@host:~ $ export PS1="[\t] \u@\h:\w\\n\\$ "
 
 [16:40:00] user@host:~
 $ 
 ```
 
 Want to control that precisely? Want to add color?
 
 There's [a guide](https://www.digitalocean.com/community/tutorials/how-to-customize-your-bash-prompt-on-a-linux-vps) for that.
 
 ---
 
 # Finding files
 
 The `find PATH` command looks through the filesystem at PATH to find files.
 
 One can then filter the output with `grep`, or use `find`'s own keys for sophisticated filtering.
 
 ```
 user@host:~ $ find ~/scits-training -name '*.sh'
 /home/user/scits-training/scripts/boom.sh
 /home/user/scits-training/scripts/beer.sh
 ```
 
 ---
 
 # `xargs`
 
 The `xargs` command can be used to convert output into arguments of another command.
 
 `xargs COMMAND` will take input and pass it as separate arguments after `COMMAND`:
 
 ```
 user@host:~ $ find ~/scits-training -name '*.sh' | xargs cat
 [contents of both .sh files]
 ```
 
 This is equivalent to 
 ```
 user@host:~ $ cat /home/user/scits-training/scripts/boom.sh \
 /home/user/scits-training/scripts/beer.sh
 ```
 
 ---
 
 # Public key authentication: theory
 
 It can be useful to use key authentication instead of standard password authentication.
 
 * Far more secure — suitable for internet-facing computers.
 * May be required in cloud environments to set new VMs.
 * Allows passwordless authentication for more convenience.
 
 --
 
 It is based on modern cryptography and consists of pairs of keys: .highlight[public],
   which you can give to others, and .highlight[private], that you keep yourself
   (preferably encrypted with a .highlight[passphrase]).
 
 Having the private key allows you to prove that you own the keypair to anyone having
   your public key, without disclosing the private key itself.
 
 ---
 
 # Public key authentication: workflow
 
 When setting up public key authentication on Linux, here's the workflow:
 
 1. You generate a keypair: private key and public key files.
 
   On Linux/Mac, `ssh-keygen` is used. On Windows, PuTTYgen can be used.
 
 2. You copy the public to the remote system.
 
 3. You connect, instructing SSH to use the private key.
 
   If it's encrypted, you're asked for the passphrase (and may be cached in an SSH agent after that).
 
 ---
 
 # Public key authentication: workflow
 
 On most Linux systems, it is sufficient to have your public key in `~/.ssh/authorized_keys` file.
 
 To copy the keypair to the remote system, `ssh-copy-id` script can be used.
 
 --
 
 Otherwise, you need to create that file/folder yourself, and make sure they have proper permissions.
 
 ```
 user@remote:~$ ls -l ~/.ssh
 total 12
 drwx------  2 user user 4096 Sep 13  2017 .
 drwxr-xr-x 13 user user 4096 Sep  2 21:47 ..
 -rw-------  1 user user 1159 Jan 21  2018 authorized_keys
 user@remote:~$ cat .ssh/authorized_keys
 ssh-rsa AAAAB3NzaC1yc2EAAA[...]j6aKfAUoXOE= some comment
 ```
 
 --
 
 To use a key, one can use `-i PRIVATE_KEY_FILE` flag for `ssh`, or Pageant on Windows.

- \ No newline at end of file