Working with Containers

diff --git a/assets/singularity-workflow.png b/assets/singularity-workflow.png new file mode 100644 index 0000000..bb0a444 Binary files /dev/null and b/assets/singularity-workflow.png differ diff --git a/containers.html b/containers.html new file mode 100644 index 0000000..dc12941 --- /dev/null +++ b/containers.html @@ -0,0 +1,963 @@ + + + + Working with Containers + + + + + +

+
+class: center, middle
+
+# Working with Containers
+
+Please open this in your browser to follow along:
+
+https://goo.gl/hhkKSP
+
+---
+
+# Agenda
+
+1. The problem we're solving
+2. Virtual machines vs containers
+3. History of contaners
+4. Docker vs Singularity
+5. Singularity workflow
+6. Installing and testing Singularity
+7. Writing a Singularity definition file
+8. Using host resources
+9. Distributing Singularity containers
+
+---
+
+class: center, middle
+
+# The Problem
+
+---
+
+# Problem (for developers)
+
+Suppose you're writing some software. It works great on your machine.
+
+However, eventually it has to leave your machine: has to run on your colleague's machine,
+  or deployed in its production environment.
+
+It can be a completely different flavour of OS, with a different set of libraries
+  and supporting tools.
+
+It can be difficult to test if you accounted for all those variation
+  on your own development system. You may have things in your environment
+  you're not even aware of that make a difference.
+
+Your users could also be less technically inclined to deal with dependencies.
+  You may wish to decrease this friction.
+
+---
+
+# Problem (for users)
+
+Suppose you want to run some piece of software.
+
+First-off, you really would like some sort of turn-key solution.
+
+The build instuctions indicate 5-years-old out of date libraries
+  on top of a similarly old OS distribution.
+
+And no, the original developer is most certainly no longer available.
+
+You also don't trust this software fully not to mess up your OS.
+
+Or, you want to run it on a remote server for which you don't
+  even have the privileges to comfortably install all the dependencies.
+
+---
+
+# Problem (for researchers)
+
+Suppose you have a piece of scientific software you used
+  to obtain some result.
+
+Then someone half across the globe tries to reproduce it,
+  and can't get it to run, or worse - is getting different
+  results for the same inputs. What is to blame?
+
+Or, even simpler: your group tries to use your software a couple
+  of years after you left, and nobody can get it to work.
+
+For a reproducible way to do science with the help of software
+  packaging just the source code might not be enough;
+  the environment should also be predictable.
+
+---
+
+# Problem (for server administrators)
+
+Suppose have a hundred of users, each requesting certain software.
+
+Some of it needs to be carefully built from scratch,
+  as there are no prebuilt packages.
+
+Some of the software works with mutually-incompatible library versions.
+  Possibly even known-insecure ones.
+
+Any such software change has to go be injected in
+  a scheduled maintenance window, but users want it yesterday.
+
+And finally, _you most certainly don't_ trust any of this software
+  not to mess up your OS. You've been there before.
+
+---
+
+# What would be a solution?
+
+* A turnkey solution
+
+  A recipe that can build a working instance of your software, reliably and fast.
+
+* BYOE: Bring Your Own Environment
+
+  A way to capture the prerequisites and environment together with the software.
+
+* Mitigate security risks
+
+  Provide a measure of isolation between the software running on a system.
+  No security is perfect, but some is better than none.
+
+---
+
+class: center, middle
+
+# The Solution(s)
+
+---
+
+# Solution: Virtual Machines?
+
+A virtual machine is an isolated instance of a .highlight[whole other "guest" OS] running
+  under your "host" OS.
+
+A .highlight[hypervisor] is responsible for handling the situations where this isolation
+  causes issues for the guest.
+
+From the point of view of the guest, it runs under its own, dedicated hardware.
+  Hence, it's called .highlight[hardware-level virtualization].
+
+Most<sup>*</sup> guest/host OS combinations can run: you can run Windows on Linux,
+Linux on Windows, etc.
+
+------
+
+\* MacOS being a stinker here, of course, with their license.
+
+---
+
+# Virtual Machines: the good parts
+
+* The BYOE principle is fully realized.
+  
+  Whatever your environment is, you can package it fully, OS and everything.
+
+* Security risks are truly minimized.
+
+  Very narrow and secured bridge between the guest and the host means little
+  opportunity for a bad actor to break out of isolation
+
+* Easy to precisely measure out resources.
+
+  The contained application, together with its OS, has restricted access to
+  hardware: you measure out its disk, memory and alloted CPU.
+
+---
+
+# Virtual machines: the not so good parts
+
+* Operational overhead
+
+  For every piece of software, the full underlying OS has to be run.
+
+* Setup overhead
+
+  Starting and stopping a virtual machine is not very fast,
+  and/or requires saving its state.
+
+* Hardware availability
+
+  The isolation between the host and the guest can hinder access to
+    specialized hardware on the host system.
+
+---
+
+# Solution: Containers (on Linux)?
+
+If your host OS is Linux and your software expects Linux, there's
+  a more direct and lightweight way to reach similar goals.
+
+Recent kernel advances allow to isolate processes from the rest
+  of the system, presenting them with their own view of the system.
+
+You can package entire other Linux distributions, and with the exception
+  of the host kernel, all the userland can be different for the process.
+
+From the point of view of the application, it's running on the same hardware
+  as the host, hence containers are called
+  .highlight[operating-system-level virtualization].
+
+---
+
+# Containers: the good parts
+
+* Lower operational overhead
+
+  You don't need to run a whole second OS to run an application.
+
+* Lower startup overhead
+
+  Setup and teardown of a container is much less costly.
+
+* More hardware flexibility
+
+  You don't have to dedicate a set portion of memory to your VM
+    well in advance, or contain your files in a fixed-size filesystem.
+
+  Also, the level of isolation is up to you. You may present devices
+    on the system directly to containers if you so desire.
+
+---
+
+# Containers: the not so good parts
+
+* Kernel compatibility
+
+  Kernel is shared between the host and the container,
+    so there may be some incompatibilties.
+
+  Plus, container support is (relatively) new, so it needs a recent kernel
+    on the host.
+
+* Security concerns
+
+  The isolation is thinner than in VM case, and kernel of the host OS
+    is directly exposed. Other software is isolated at kernel level,
+    but the attack surface is larger.
+
+* Linux on Linux
+
+  Containers are inherently a Linux technology. You need a Linux host
+    (or a Linux VM) to run containers, and only Linux software can run.
+
+
+---
+
+# History of containers
+
+The idea of running an application in a different environment is not new to UNIX-like systems.
+
+Perhaps the first effort in that direction is the `chroot` command and concept (1982):
+  presenting applications with a different view of the filesystem (a different `/`, root directory).
+
+This minimal isolation was improved in in FreeBSD with `jail` (2000),
+  separating other resources (processes, users)
+  and restricting how applications can interact with each other and the kernel.
+
+Linux developed facilities for isolating and controlling access to some processes
+  with namespaces (2002) and cgroups (2007).
+
+Those facilities led to creation of solutions for containerization, notably LXC (2008),
+  Docker (2013) and Singularity (2016).
+
+---
+
+# Docker
+
+Docker came about in 2013 and since has been on a meteoritic rise
+  as a golden standard for containerization technology.
+
+A huge amount of tools is built around Docker to build, run,
+  orchestrate and integrate Docker containers.
+
+Many cloud service providers can directly integrate Docker containers.
+  Docker claims 26x efficiency improvement at cloud scale.
+
+Docker encourages splitting software into microservice chunks
+  that can be portably used as needed.
+
+---
+
+# Docker concerns
+
+Docker uses a pretty complicated model of images/volumes/metadata,
+  orchestrating swarms of those containers to work together,
+  and it not always very transparent with how those are stored.
+
+Also, isolation features require superuser privileges;
+  Docker has a persistent daemon running with those privileges
+  and many container operations require root as well.
+
+Both of those issues make Docker undesirable in applications
+  where you don't wholly own the computing resource - HPC environments.
+
+Out of those concerns, and out of scientific community, came Singularity.
+
+---
+
+# Singularity
+
+Singluarity is quite similar in principles to Docker. In fact,
+  it's pretty straightforward to convert a Docker container
+  to a Singularity image.
+
+Singularity uses a monolithic, image-file based approach.
+  Instead of dynamically overlaid "layers" of Docker, you have
+  a single file you can build once and simply copy over to the target system.
+
+The privilege problem was a concern from the ground-up, and solved by having a
+  `setuid`-enabled binary that can accomplish container startup - and drop
+  privileges completely as soon as practical.
+
+Privilege elevation inside a container is impossible: to be root inside,
+  you have to be root outside. And users don't need explicit root
+  access to operate containers (past, possibly, initial build).
+
+---
+
+# Singularity and HPC
+
+Thanks to the above improvements over Docker, HPC cluster operators
+  are much more welcoming to the idea of Singularity support.
+
+As a result of a joint Pipeline Interoperability project between Swiss Science IT groups,
+  we have a [set of guidelines](https://docs.google.com/document/d/1KIPS8j1IgY5nstQJDDuafFQlw1Spjvo5e6LSLJEBCM4)
+  for Singularity deployment and use by scientific community.
+
+Also as part of this effort, UBELIX will support Singularity
+  in the near future (sadly not yet).
+
+Once your software is packaged in Singularity, it should work across
+  all Science IT platforms supporting the technology.
+
+---
+
+# Singularity workflow
+
+.minusmargin[
+  ![:scale 100%](assets/singularity-workflow.png)
+]
+
+The general idea: prepare container on a local machine with full control,
+  then can execute on a resource without root.
+
+---
+
+# Installing Singularity
+
+Installing Singularity from source is probably preferred,
+  as it's still a relatively new piece of software.
+
+Instructions can be found at http://singularity.lbl.gov/install-linux
+
+On Ubuntu, `build-essential` provides enough prerequisites for compilation.
+
+.exercise[
+  Follow build instructions on your test system. You will need root access!
+]
+
+---
+
+# Using Singularity
+
+If you followed build instructions, you should now have `singularity`
+  available from the shell.
+
+```
+user@host:~$ singularity --version
+2.3.1-dist
+```
+
+The general format of Singularity commands is:
+
+```
+singularity [<global flags>] <command> [<command flags>] [<arguments>]
+```
+
+Singularity is pretty sensitive to the order of those.
+
+Use `singularity help [<command>]` to check built-in help.
+
+You can find the configuration of Singularity under `/usr/local/etc/singularity`
+
+---
+
+# Container images
+
+A Singularity image is, for practical purposes, a filesystem tree
+  that will be presented to the applications running inside it.
+
+A Docker container is built with a series of _layers_ that are
+  stacked upon each other to form the filesystem. Singularity
+  collapses those into a single, portable file.
+
+A container needs to be somehow bootstrapped to contain a base
+  operating system before further modifications can be made.
+
+---
+
+# Pulling Docker images
+
+The simplest way of obtaining a working Singularity image
+  is to pull it from either Dockerhub or Singularity Hub.
+
+Let's try it with CentOS 6:
+
+```
+user@host:~$ singularity pull docker://centos:6
+```
+
+This will download the layers of the Docker container to your
+machine and assemble them into an image.
+
+---
+
+# Pulling Docker images
+
+```
+user@host:~$ singularity pull docker://centos:6
+Initializing Singularity image subsystem
+Opening image file: centos-6.img
+Creating 333MiB image
+Binding image to loop
+Creating file system within image
+Image is done: centos-6.img
+Docker image path: index.docker.io/library/centos:6
+Cache folder set to /home/ubuntu/.singularity/docker
+[1/1] |===================================| 100.0%
+Importing: base Singularity environment
+Importing: /home/ubuntu/.singularity/docker/sha256:cd3b990dbbea1a8de88713ad75b277692a0d5ffe428c342b53d12d6b82f10494.tar.gz
+Importing: /home/ubuntu/.singularity/metadata/sha256:a43b5b7bed1c383b6a89186257392f8630c8c26b5e95f7edf92e4b788a73383e.tar.gz
+Done. Container is at: centos-6.img
+```
+
+Note that this .highlight[does not require `sudo` or Docker]!
+
+.exercise[
+  Pull the CentOS 6 image from Dockerhub with the above command
+]
+
+---
+
+# Entering shell in the container
+
+To test our freshly-created container, we can invoke an interactive shell
+  to explore it with .highlight[`shell`]:
+
+```
+user@host:~$ singularity shell centos-6.img
+Singularity: Invoking an interactive shell within container...
+
+Singularity centos-6.img:~>
+```
+
+At this point, you're within the environment of the container.
+
+We can verify we're "running" CentOS:
+
+```
+Singularity centos-6.img:~> cat /etc/centos-release
+CentOS release 6.9 (Final)
+```
+
+---
+
+# User/group within the container
+
+Inside the container, we are the same user:
+
+```
+Singularity centos-6.img:~> whoami
+user
+```
+
+We will also have the same groups. That way, if any host resources are mounted
+  in the container, we'll have the same access privileges.
+
+If we launched `singularity` with `sudo`, we would be `root` inside the container.
+
+---
+
+# Default mounts
+
+Additionally, our home folder, and the folder we've invoked Singularity from,
+are accessible inside the container (by default):
+```
+Singularity centos-6.img:~> ls ~
+[..lists home folder..]
+```
+
+We have access to the bound folders with the same rights as outside,
+  so we can e.g. write to 
+
+```
+Singularity centos-6.img:~> touch ~/test_container
+Singularity centos-6.img:~> exit
+user@host:~$ ls ~/test_container
+/home/user/test_container
+```
+
+The current working directory inside the container is the same
+  as outside at launch time.
+
+---
+
+# Running a command directly
+
+Besides the interactive shell, we can execute any command
+  inside the container directly with .highlight[`exec`]:
+
+```
+user@host:~$ singularity exec centos-6.img cat /etc/centos-release
+CentOS release 6.9 (Final)
+```
+
+.exercise[
+  Invoke the `python` interpreter with `exec`.
+]
+
+---
+
+# STDIO with container processess
+
+Standard input/output are processed as normal by Singularity.
+You can redirect them:
+
+```
+ubuntu@host:~$ singularity exec centos-6.img echo Boo! > ~/test_container
+ubuntu@host:~$ singularity exec centos-6.img cat < ~/test_container
+Boo!
+```
+
+You can use containers in pipelines:
+
+```
+$ singularity exec centos-6.img echo Boo! | singularity exec centos-6.img cat
+Boo!
+```
+
+.exercise[
+  Count the number of words in host's `ls /etc`'s output using container's copy of `wc`,
+  then the other way around. Hint:
+
+  ```
+  ls /etc | wc -w
+  ```
+]
+
+---
+
+# Modifying the container
+
+In order to modify the container, we need .highlight[`-w` (write access) and root].
+
+```
+user@host:~$ sudo singularity shell -w centos-6.img
+Singularity centos-6.img:~> whoami
+root
+```
+
+Let's get a missing application, `fortune`.
+It's not available in the default distribution,
+we will need to enable a community repository and install it:
+
+```
+Singularity centos-6.img:~> yum -y --enablerepo=extras install epel-release
+Singularity centos-6.img:~> yum -y install fortune-mod
+```
+
+.exercise[
+  Follow those steps, exit the container, and try out `fortune` with `exec`.
+]
+
+---
+
+# Resizing the container
+
+It can happen that the size of the container at creation is insufficient.
+
+If we try to install `vim` in the container, we'll be met with errors:
+
+```
+user@host:~$ sudo singularity exec -w centos-6.img yum install -y vim
+[..output..]
+Error Summary
+-------------
+Disk Requirements:
+  At least 43MB more space needed on the / filesystem.
+```
+
+We can resize the container with .highlight[`expand`]. Let's add 100 MiB:
+```
+user@host:~$ singularity expand --size 100 centos-6.img
+[..output..]
+user@host:~$ sudo singularity exec -w centos-6.img yum install -y vim
+[..output..]
+Complete!
+```
+
+---
+
+# Giving container purpose
+
+A container can have a "default" command which is run without specifying it.
+
+Inside the container, it's `/singularity`. Let's try modifying it:
+
+```
+user@host:~$ sudo singularity exec -w centos-6.img vim /singularity
+```
+
+By default you'll see the following:
+
+```bash
+#!/bin/sh
+
+exec /bin/bash "$@"
+```
+
+This is a script that will pass all arguments to `/bin/bash`.
+
+---
+
+# Giving container purpose
+
+We installed `fortune`, so let's use that instead:
+
+```bash
+#!/bin/sh
+
+exec /usr/bin/fortune "$@"
+```
+
+.exercise[
+  Make the same modification to your container.
+]
+
+Now we can invoke it with .highlight[`run`] or even by
+  .highlight[running the image]:
+
+```
+$ singularity run centos-6.img
+[..some wisdom or humor..]
+$ ./centos-6.img
+[..some more wisdom or humor..]
+```
+
+---
+
+# Making the container reproducible
+
+Instead of taking some base image and making changes to it by hand,
+  we want to make this build process reproducible.
+
+This is achieved with .highlight[definition files].
+
+Let's try to retrace out steps to obtain a fortune-telling CentOS.
+
+.exercise[
+  Open a file called `fortune.def` in an editor, and prepare to copy along
+]
+
+---
+
+# Bootstrapping
+
+The definition file starts with a header section.
+
+The key part of it is the `Bootstrap:` configuration.
+
+There are 2 currently types of bootstrap methods:
+
+* using `yum`/`apt`/`pacman` on the host system to bootstrap a similar one
+* pull a Docker image
+
+We'll be using the Docker method.
+
+```
+Bootstrap: docker
+From: centos:6
+```
+
+---
+
+# Setting up the container
+
+There are 2 sections for setup commands (essentially shell scripts):
+
+1. .highlight[`%setup`] for commands to be executed .highlight[outside the container].
+
+  You can use `$SINGULARITY_ROOTFS` to access the container's filesystem,
+    as it is mounted on the host during the build.
+
+2. .highlight[`%post`] for commands to be executed .highlight[inside] the container.
+
+  This is a good place to set up the OS, such as installing packages.
+
+---
+
+# Setting up the container
+
+Let's save the name of the build host and install `fortune`:
+
+```
+Bootstrap: docker
+From: centos:6
+
+%setup
+  hostname -f > $SINGULARITY_ROOTFS/etc/build_host
+
+%post
+  yum -y --enablerepo=extras install epel-release
+  yum -y install fortune-mod
+```
+
+---
+
+# Adding files to the container
+
+An additional section, .highlight[`%files`], allows to copy files or folders to the container.
+
+We won't be using it here, but the format is as follows (like `cp`, but destination is inside):
+
+```
+%files
+  some/file /some/other/file some/path/
+  some/directory some/path/
+```
+
+Note that this happens _after_ `%post`. If you need the files earlier, copy them manually in `%setup`.
+
+---
+
+# Setting up the environment
+
+You can specify a script to be sourced when something is run in the container.
+
+This goes to the .highlight[`%environment`] section. Treat it like `.bash_profile`.
+
+```
+%environment
+  export HELLO=World
+```
+
+Note that the host environment variables are passed on by default as well, unless `-e` is specified.
+
+---
+
+# Setting up the runscript
+
+The runscript (`/singularity`) is specified in the `%runscript` section.
+
+Let's use the file we copied at `%setup` and run `fortune`:
+
+```
+%runscript
+  read host < /etc/build_host
+  echo "Hello, $HELLO! Fortune Teller, built by $host"
+  exec /usr/bin/fortune "$@"
+```
+
+---
+
+# Testing the built image
+
+You can specify commands to be run at the end of the build process inside the container
+  to perform sanity checks.
+
+Use `%test` section for this:
+
+```
+%test
+  test -f /etc/build_host
+  test -f /usr/bin/fortune
+```
+
+All commands must return successfully or the build will fail.
+
+---
+
+# The whole definition file
+
+```
+Bootstrap: docker
+From: centos:6
+
+%setup
+  hostname -f > $SINGULARITY_ROOTFS/etc/build_host
+%post
+  yum -y --enablerepo=extras install epel-release
+  yum -y install fortune-mod
+%environment
+  export HELLO="World"
+%runscript
+  read host < /etc/build_host
+  echo "Hello, $HELLO! Fortune Teller, built by $host"
+  exec /usr/bin/fortune "$@"
+%test
+  test -f /etc/build_host
+  test -f /usr/bin/fortune
+```
+
+.exercise[
+  Check that your `fortune.def` is the same as above.
+]
+
+---
+
+# Building a container from definition
+
+To fill a container using a definition file, we must first .highlight[create] it.
+
+This is the time to specify the image size:
+
+```
+user@host:~$ singularity create -s 512 fortune.img
+```
+
+Then, .highlight[bootstrap] using the definition (.highlight[requires root]):
+
+```
+user@host:~$ sudo singularity bootstrap fortune.img fortune.def
+```
+
+.exercise[
+1. Bootstrap the image as shown above.
+
+2. Test running it directly.
+]
+
+---
+
+# Host resources
+
+A Singularity container can have more host resources exposed.
+
+For providing access to more directories,
+  one can specify bind options at runtime with `-B`:
+
+```
+-B source[:destination[:mode]]
+```
+
+where .highlight[source] is the path on the host,
+  .highlight[destination] is the path in a container (if different)
+  and .highlight[mode] is optionally `ro` if you don't want to give write access.
+
+Additionally, devices on the host can be exposed, e.g. the GPU; but you need to make
+  sure that the guest has the appropriate drivers. One solution is to bind the drivers on the container.
+
+OpenMPI should also work, provided the libraries on the host and in the container are sufficiently close.
+
+---
+
+# Fuller isolation
+
+By default, a container is allowed a lot of "windows" into the host system (dictated by Singularity configuration).
+
+For an untrusted container, you can further restrict this with options like `--contain`, `--containall`.
+
+In this case, you have to manually define where standard binds like the home folder or `/tmp` point.
+
+See `singularity help run` for more information.
+
+---
+
+# Distributing the container
+
+Using the container after creation on another Linux machine is simple:
+  you simply copy the image file there.
+
+Note that you can't just run the image file on a host without Singularity installed!
+
+.exercise[
+  Test the above, by trying to run `fortune.img` inside itself.
+]
+
+You can easily integrate Singularity with the usual scheduler scripts (e.g. Slurm).
+
+---
+
+# Using Singularity Hub
+
+Singularity Hub allows you to cloud-build your containers from Bootstrap files,
+  which you can then simply `pull` on a target host.
+
+This requires a GitHub repository with a `Singularity` definition file.
+
+After creating an account and connecting to the GitHub account,
+  you can select a repository and branches to be built.
+
+Afterwards, you can pull the result:
+
+```
+$ singularity pull shub://kav2k/fortune
+$ ./kav2k-fortune-master.img
+Hello, World! Fortune Teller, built by shub-builder-103338658-2826.c.srcc-gcp-ruth-will-phs-testing.internal
+```
+
+---
+
+# Docker and Singularity
+
+Instead of writing a Singularity file, you may write a `Dockerfile`,
+  build a Docker container and convert that.
+
+Pros:
+
+* More portable: for some, using Docker or some other container solution is preferable.
+
+* Easier private hosting: Singularity Hub doesn't seem to currently support private instances.
+
+Cons:
+
+* Blackbox: Singularity understands less about the build process, in terms of container metadata.
+
+* Complexity: Another tool to learn if you don't know Docker already.
+
+---
+
+# Docker -> Singularity
+
+If you have a Docker image you want to convert to Singularity,
+  you have at least two options:
+
+1. Upload the image to a Docker Registry (such as Docker Hub)
+  and `pull` from there.
+
+2. Convert locally with Docker and `docker2singularity`
+  https://github.com/singularityware/docker2singularity
+
+---
+
+# Runner script
+
+We have developed a small runner script that allows to specify
+  a lot of "external" metadata (like bind commands or image size)
+  in a single configuration file alongside other bootstrap files,
+  as well as scripts to run on the host to run/validate the container.
+
+You can install it with `pip install singularity-pipeline`
+
+Documentation and samples available at https://c4science.ch/diffusion/2915/browse/master/UniBe/
+
+Consider it still in alpha, but it helps writing down the metadata
+  for a scientific pipeline.
+
+

+ + + + + + + \ No newline at end of file diff --git a/containers.zip b/containers.zip new file mode 100644 index 0000000..249be1a Binary files /dev/null and b/containers.zip differ diff --git a/css/common.css b/css/common.css index 276577e..c69531e 100644 --- a/css/common.css +++ b/css/common.css @@ -1,125 +1,130 @@ @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz); @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic); @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic); body { font-family: 'Droid Serif'; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: normal; } .remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; } .remark-code-line .hljs-constant { color: #5BFF68 } .remark-slide-content { font-size: 24px; } table.stack { border-collapse: collapse; table-layout: fixed; width: 100%; margin: auto; } .stack tr td:first-child { border: 1px black solid; height: 3em; width: 60%; } .stack .bluebg { background: SkyBlue } table.quiz { width: 100%; margin: auto; } .quiz td { border: 1px black solid; } .quiz .clicked.Y { background: #CCFFCC; } .quiz .clicked.N { background: #FFCCCC; } .two-columns table { table-layout: fixed; } .two-columns td { width: 50%; } .mnemonic { border: 1px solid firebrick; border-radius: 0.3em; margin-bottom: 0.5em; } .mnemonic::before { content: "Mnemonic:"; margin-left: 0.25em; font-style: italic; color: firebrick; } .mnemonic p { margin: 0.5em; } .exercise { border: 1px solid blue; border-radius: 0.3em; margin-bottom: 0.5em; } .exercise::before { content: "Exercise:"; margin-left: 0.25em; font-style: italic; color: blue; } .exercise p { margin: 0.5em; } .highlight { color: red; font-weight: bold; } .middle img { vertical-align: middle; } .small { font-size: smaller; } +.minusmargin { + margin-left: -10%; + margin-right: -10%; +} + div.blinking { -webkit-animation: 1s blink step-end infinite; -moz-animation: 1s blink step-end infinite; -ms-animation: 1s blink step-end infinite; -o-animation: 1s blink step-end infinite; animation: 1s blink step-end infinite; display: inline-block; } @-webkit-keyframes blink { 0% { opacity: 1.0; } 50% { opacity: 0.0; } 100% { opacity: 1.0; } } @keyframes blink { 0% { opacity: 1.0; } 50% { opacity: 0.0; } 100% { opacity: 1.0; } } \ No newline at end of file diff --git a/js/terminal.language.js b/js/terminal.language.js index 9e9b282..996ddfd 100644 --- a/js/terminal.language.js +++ b/js/terminal.language.js @@ -1,26 +1,26 @@ /* Language: terminal console Author: Josh Bode */ var hljs = remark.highlighter.engine; hljs.registerLanguage('terminal', function() { return { contains: [ { className: 'string', - begin: '^(([\\w.]+)@([\\w.]+)\\:(\\S+) )?\\$' + begin: '^(([\\w.]+)@([\\w.]+)\\:(\\S+) ?)?\\$' }, /*{ className: 'constant', begin: '(:(.*) )?\\$ ' },*/ { className: 'ansi', begin: '', end: '<\\/span>' } ] } }); diff --git a/linux_1.zip b/linux_1.zip new file mode 100644 index 0000000..05827f2 Binary files /dev/null and b/linux_1.zip differ diff --git a/linux_2.html b/linux_2.html index 4e352ce..213b6d7 100644 --- a/linux_2.html +++ b/linux_2.html @@ -1,1147 +1,1259 @@ - Introduction to Linux, Part 1 + Introduction to Linux, Part II

class: center, middle
 
 # Introduction to Linux
 
 ## Part II
 
 https://scits.math.unibe.ch/linux_2.html
 
 ???
 
 Notes for the _first_ slide!
 
 ---
 
 # Agenda
 
 1. Linux resources you can use
 2. Standard input/output and its redirection
 3. UNIX pipelines
 4. Background processes 
 5. Durable sessions with `screen`
 6. File ownership and permissions
 7. Shell scripting basics
 8. Customizing your environment
 
 ---
 
 # What Linux resources can I use?
 
 To do development and run light workloads:
 
 * Your own computer may already run Linux.
 
 * You can install Linux in a virtual machine.  
   I recommend Virtualbox.
 
 * If you're running Windows 10, you can install  
   [Windows Subsystem for Linux](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux)
 
 ---
 
 # What if it's not enough?
 
 To create persistent services:
 
 * Ask your group's sysadmin for servers/VM resources.
 
 * UniBe Informatikdienste offers virtual machines.
 
 * Cloud resources: [SWITCHengines](https://www.switch.ch/engines/), other cloud services.
 
 To run heavy calculations:
 
 * [UBELIX Linux cluster](https://docs.id.unibe.ch/ubelix/ubelix-101).
 
 * Your group may have in-house infrastructure.
 
 * Again, cloud services.
 
 ---
 
 # Prepare for the tutorial
 
 You should be either running Linux, or connected to a Linux system.
 
 Log in to UBELIX, or inform me that you need access to another system.
 
-Some later steps require a Linux machine you control.
-
 Execute the following to set up training (if you haven't already):
 ```
 $ wget https://scits.math.unibe.ch/script -O - | /bin/bash
 ```
 
 ---
 
 # Processes and their input/output
 
 When you're connected to a Linux system,
 what you normally see is the shell prompt, awaiting input:
 
 ```
 user@host:~ $
 ```
 
 A .highlight[process] called shell is responsible for input/output at this moment.
 
 By default, the input is either a keyboard connected to the system,
   or your keypresses being relayed over the network.
 
 The output, by default, is the screen connected to the system,
   or text being relayed for display over the network.
 
 ---
 
 # Processes and their input/output
 
 Those are called .highlight[standard input] and .highlight[standard output],
   or .highlight[STDIN/STDOUT].
 
 .center[
   ![](assets/shell_stdio.svg)
 ]
 
 --
 
 If the user issues a command that calls another program, the shell
   creates a .highlight[child process] and attaches the input/output to it.
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 The shell will wait until the program terminates,
   after which STDIN/STDOUT get reattached and a prompt is displayed.
 
 ---
 
 # Redirecting output
 
 Sometimes, we want to capture what a command is outputting to the screen.
 
 For example, suppose that we want to save into a file the directory structure returned by `ls -R`:
 
 ```
 user@host:~ $ ls -R ~/scits-training
 scits-training:
-animals  moving  numbers  io
+animals  moving  numbers  io  scripts
 [...]
 user@host:~ $
 ```
 
 We can instruct the shell to .highlight[redirect] the standard output:
 
 .center[
   ![](assets/stdout_redirect.svg)
 ]
 
 ---
 
 # Redirecting output
 
 To redirect the output, we add .highlight[`> FILE`] to the command:
 
 ```
 user@host:~ $ cd ~/scits-training/io
 user@host:~/scits-training/io $ ls -R ~/scits-training > listing
 user@host:~/scits-training/io $ cat listing
 scits-training:
-animals  moving  numbers  io
+animals  moving  numbers  io  scripts
 [...]
 ```
 
 This will overwrite the contents of `FILE` (if any) with the output of the command.
 
 The file will be created, if it does not exist yet.
 
 ---
 
 # Appending output
 
 Sometimes we don't want to overwrite (sometimes called "clobber") the file
   with new contents and add them to the end instead.
 
 To do that, use .highlight[`>> FILE`] instead of `> FILE`.
 
 .exercise[
 1. Try running the command `date` to see what it outputs.
 
 2. Run `date` 3 times, appending the output to a file `date.log`.
 
 3. Verify with `cat` that the file contains 3 records.
 ]
 
 ---
 
 # Error output
 
 Try saving the output of a command with errors
 and you'll see that it still outputs to the screen:
 
 ```
 user@host:~/scits-training/io $ ls , > listing
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $
 ```
 
 --
 
 This is intentional: Linux actually has two output streams for
   its command line, STDOUT for normal data and .highlight[STDERR] for errors.
 
 .center[
   ![](assets/stderr.svg)
 ]
 
 This simplifies debugging: errors are separate from data.
 
 ---
 
 # Error output
 
 STDERR is not redirected when using `>` or `>>`:
 
 .center[
   ![](assets/stderr_noredirect.svg)
 ]
 
 --
 
 It's possible to redirect it as well with .highlight[`2>`] or .highlight[`2>>`]:
 
 ```
 user@host:~/scits-training/io $ ls .. , > listing 2> errors
 user@host:~/scits-training/io $ cat errors
 ls: cannot access ,: No such file or directory
 user@host:~/scits-training/io $ cat listing
 ..:
-animals  io  moving  numbers
+animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Discarding output
 
 Sometimes we don't need output at all.
 
 In this case, we can redirect it to a special file, .highlight[`/dev/null`]
 
 It's a .highlight[device] that will accept any input, discarding it immediately.
 
 --
 
 For example, one might want to silence errors:
 
 ```
 user@host:~/scits-training/io $ ls .. , 2> /dev/null
 ..:
-animals  io  moving  numbers
+animals  io  moving  numbers  scripts
 ```
 
 ---
 
 # Interactive input
 
 Most commands we've seen don't require any interactive input.
 
 `tr` (for .highlight[translate]) is a command that transforms its input:
   it substitutes some characters with others.
 
 For example, `tr 'a-z' 'A-Z'` would translate all lowercase letters into uppercase.
 
 --
 
 Let's try this:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 ```
 
 ---
 
 # Interactive input
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z'
 Let's input some text
 LET'S INPUT SOME TEXT
 But enter doesn't stop it!
 BUT ENTER DOESN'T STOP IT!
 ```
 
 A problem: text can contain many lines, and the program won't know when to stop.
 
 --
 
 We can terminate the program with Ctrl+C, but it actually expects an .highlight[end of input].
 
 We can signal end of input with Ctrl+D on an empty line (or pressing it twice).
 
 --
 
 .exercise[
   What happens if we press Ctrl+D while back at the shell prompt?
 ]
 
 ---
 
 # Redirecting input
 
 What if we want to use a file as an input in a command that doesn't accept files as arguments,
   we need instruct the shell to use the file as the program's standard input:
 
 .center[
   ![](assets/stdin_redirect.svg)
 ]
 
 This is done with adding .highlight[`< FILE`] to the command.
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' < errors
 LS: CANNOT ACCESS ,: NO SUCH FILE OR DIRECTORY
 ```
 
 ---
 
 # Redirects
 
 .exercise[
 1. Combine input and output redirection to save the output of last `tr` command into `errors.uppercase`
 
 2. Use `cat` to verify the saved output.
 ]
 
 ---
 
 # Pipelines
 
 We have shown how to save outputs to a file, and further process files as inputs.
 
 Sometimes, we don't need to save this intermediate representation.
 In that case, we can directly connect the output of one program to the input of another
 with .highlight[pipes].
 
 To do so, separate two commands with .highlight[`|`]:
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 LISTING
 ```
 ---
 
 # Pipelines
 
 ```
 user@host:~/scits-training/io $ ls . | tr 'a-z' 'A-Z'
 DATE.LOG
 ERRORS
 LISTING
 ```
 
 Given this command, shell starts two processes in parallel and ties their respective output and input together.
 
 Standard input/output is connected at the ends of the chain:
 
 .center[
   ![](assets/pipe.svg)
 ]
 
 --
 
 Such .highlight[pipelines] can be longer than two commands, and can be combined with file redirects.
 
 ---
 
 # Pipelines and errors
 
 You will notice that all errors are still output normally:
 
 ```
 user@host:~/scits-training/io $ ls . , | tr 'a-z' 'A-Z'
 ls: cannot access ,: No such file or directory
 .:
 DATE.LOG
 ERRORS
 LISTING
 ```
 
 --
 
 As before, errors are not normally redirected, and collected from all processes in the pipe:
 
 .center[
   ![](assets/pipe_err.svg)
 ]
 
 ---
 
 # Background jobs
 
 Recall that, when running a command, the shell waits until it is terminated: all input goes to the program (or nowhere).
 
 .center[
   ![](assets/child_stdio.svg)
 ]
 
 Sometimes, we don't need to wait until the program terminates – we actually want it running in background.
 
 ---
 
 # Background jobs
 
 If you specify .highlight[`&`] at the end of the command, the shell will start it, but keep control of STDIN:
 
 .center[
   ![](assets/job_stdio.svg)
 ]
 
 Instead of a .highlight[foreground] process, it becomes a .highlight[background] job.
 
 --
 
 You are immediately returned to the shell and can run other commands while the job executes.
 
 Note that both the shell and the background job are connected to STDOUT.
 Redirect output to prevent mix-ups.
 
 ---
 
 # Background jobs
 
 Compare:
 
 ```
 user@host:~/scits-training/io $ sleep 3
 user@host:~/scits-training/io $ sleep 3 &
 [1] 12231
 user@host:~/scits-training/io $
 ```
 
 Here, `[1]` is the .highlight[job number], and `12231` is the .highlight[process ID], or PID.
 
 After 3 seconds and when another command finishes (you can just press Enter for an empty command),
 you'll be informed that the job terminated:
 
 ```
 user@host:~/scits-training/io $
 [1]+  Done                    sleep 3
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Listing jobs
 
 You can list running background jobs with .highlight[`jobs`]:
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12232
 user@host:~/scits-training/io $ sleep 0 &
 [2] 12233
 user@host:~/scits-training/io $ jobs
 [1]-  Running                 sleep 100 &
 [2]+  Done                    sleep 0
 ```
 
 ---
 
 # Terminating jobs
 
 You can forcibly .highlight[terminate] a job with the .highlight[`kill`] command,
   which accepts either PID or job ID (with `%`):
 
 ```
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12234
 user@host:~/scits-training/io $ kill 12234
 user@host:~/scits-training/io $ jobs
 [1]+  Terminated              sleep 100
 
 user@host:~/scits-training/io $ sleep 100 &
 [1] 12235
 user@host:~/scits-training/io $ kill %1
 ```
 
+You can search for more process IDs to terminate with .highlight[`ps ax`], in case something is misbehaving.
+
 ---
 
 # Stopped jobs
 
 Background jobs have nothing connected to their standard input.
 
 If a background job cannot continue without user input, it will .highlight[stop],
 which the shell will signal to you:
 
 ```
 user@host:~/scits-training/io $ tr 'a-z' 'A-Z' &
 [1] 12236
 user@host:~/scits-training/io $
 [1]+  Stopped                 tr /a-z/ /A-Z/
 user@host:~/scits-training/io $
 ```
 
 --
 
 You can bring a job to .highlight[foreground] to pass STDIN from the shell
 to the running job with `fg` (or `fg %N` for a specific job number):
 
 ```
 user@host:~/scits-training/io $ fg
 tr /a-z/ /A-Z/
 You are now talking to the job
 YOU ARE NOW TALKING TO THE JOB
 ```
 
 ---
 
 # Stopping and resuming programs
 
 You can stop most currently-running programs with `Ctrl`+`Z`:
 
 ```
 user@host:~/scits-training/io $ sleep 100
 ^Z
 [1]+  Stopped                 sleep 100
 user@host:~/scits-training/io $
 ```
 
 --
 
 From there, you can use `fg` to resume normal execution of the program,
   or use .highlight[`bg`] to let it continue to run in the background.
 
 ```
 user@host:~/scits-training/io $ bg
 [1]+ sleep 100 &
 user@host:~/scits-training/io $ jobs
 [1]+  Running                 sleep 100 &
 ```
 
 ---
 
 # Background jobs are fragile
 
 What will happen if you start a background job, and then close the terminal?
 
 --
 
 Closing the terminal (or disconnecting the SSH session) kills the shell you were talking to.
 Since the job was a .highlight[child process] of that shell, it will also be killed.
 
 --
 
 A minor inconvenience if you're working on your own machine (you can just leave the terminal open),
 but a much bigger problem with remote connections.
 
 If the connection is broken, the shell is also terminated along with all processes launched from it.
 
 How to protect against it?
 
 ---
 
 # `screen`
 
 To protect your session, you can use .highlight[`screen`].
 
 `screen` starts a new shell that exists independently of your current one.
 
 Even if the current shell dies (e.g. because you disconnected),
 the shell running in `screen` will continue together with all its child processes.
 
 --
 
 Starting a new `screen` session is simple:
 
 ```
 user@host:~/scits-training/io $ screen
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen!"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Reattaching to `screen`
 
 Now suppose your connection was terminated.
 
 Close the terminal where it is running to simulate that, then log in again.
 
 --
 
 You can use .highlight[`screen -ls`] to list active sessions:
 
 ```
 user@host:~ $ screen -ls
 There is a screen on:
         13383.pts-2.host        (11/09/17 03:02:23)     (Detached)
 1 Socket in /var/run/screen/S-user.
 ```
 
 --
 
 You can attach to a screen session (possibly detaching it first, if it's being used somewhere)
   with .highlight[`-dR`] (for .highlight[detach, reattach])
 
 ```
 user@host:~ $ screen -dR
 [terminal screen is cleared]
 user@host:~/scits-training/io $ echo "Hello, I'm in a screen!"
 Hello, I'm in a screen!
 user@host:~/scits-training/io $
 ```
 
 ---
 
 # Controlling screen
 
 `screen` can be used for other things, such as having multiple parallel shell sessions open.
 
 Controlling screen consists of pressing .highlight[`Ctrl`+`A`], then a screen-specific command.
 
 For example,
 
 * .highlight[`c`] will .highlight[create] a new shell within screen
 * .highlight[`n`] will switch to the .highlight[next] shell
 * .highlight[`d`] will .highlight[detach] from `screen`, returning you to the original shell
 
 Finally, you can use .highlight[`?`] to access built-in help,
 or use `man screen` for a more detailed manual.
 
 ---
 
 # Users and groups
 
 Before we discuss permissions, we need to understand users and groups in Linux.
 
 A .highlight[user] is a unit of access control; it has a set of credentials to access the system
   and .highlight[owns] some files on it.
 
 A .highlight[group] is a collection of users to facilitate shared access to resources.
   A user can belong to many groups but one group is considered primary.
 
 You can use `id` to check your user and groups:
 
 ```
 akashev@submit01:~ $ id
 uid=7265(akashev) gid=1109(math) groups=1109(math),902(l_gaussian)
 ```
 
 Here, `akashev` is my user, `math` is my primary group and `l_gaussian` is another group I belong to.
 
 ---
 
 # Permissions: `rwx`
 
 Each file and directory in UNIX filesystems has 3 permissions (for a particular user).
 
 Regular files:
 * .highlight[`r`], or .highlight[Read], means that you can read the contents of a file.
 * .highlight[`w`], or .highlight[Write], means that you can modify the file.
 * .highlight[`x`], or .highlight[eXecute], means that the file may be launched as a program.
 
 Directories:
 * .highlight[`r`] means that you can read the list of files within the directory.
 * .highlight[`w`] means that you can add or delete files from the directory.
 * .highlight[`x`] means you can .highlight[traverse] the folder: enter it with `cd` and read the contents of its files.
 
 ---
 
 # Inspecting permissions
 
 Try running `ls -la` to see permissions on files and folders:
 
 ```
 $ ls -la
 total 20
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 drwxrwxr-x 6 user group 4096 Sep 10 23:06 ..
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 -rw-rw-r-- 1 user group   47 Sep 11 00:50 errors
 -rw-rw-r-- 1 user group   30 Sep 11 01:09 listing
 ```
 
 We're interested in the first column: the cryptic `drwxrwxr-x` and `-rw-rwr--`,
 which are called .highlight[mode].
 
 --
 
 * The first character denotes the .highlight[file type].
   * `-` means "regular file".
   * `d` means "directory".
 * The rest is divided in groups of three:
   * Access for the owner
   * Access for the group
   * Access for everyone else
 
 ---
 
 # File ownership
 
 ```
 drwxrwxr-x 2 user group 4096 Sep 11 01:26 .
 -rw-rw-r-- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 Each file in a UNIX filesystem has an .highlight[owner] and a .highlight[group] attached.
 
 In the example above, `user` is the owner and `group` is the designated group.
 
 Note that the user .highlight[doesn't have to be in the assigned group].
 
 ---
 
 # Effective permissions
 
 ```
 -rwxr-x--- 1 user group   90 Sep 10 23:08 script
 ```
 
 To determine which permissions apply, the following is checked:
 
 * If the user is the owner, the first set applies (`rwx`, full permissions)
 * If the user is in the designated group, the second set applies (`r-x`, so cannot write)
 * For all other users, the third set applies (`---`, so cannot do anything)
 
 --
 
 A special user, .highlight[superuser] (normally called .highlight[root]),
   can completely disregard permissions and do anything to any file on the system.
 
 ---
 
 # Permissions: first match applies
 
 Note that the system does not apply "best" permissions – only the first set that matches.
 
 Let's reverse the situation:
 
 ```
 ----r-xrwx 1 user group   90 Sep 10 23:08 script
 ```
 
 For this file, the owner cannot do anything to the file, anyone in `group` cannot modify it,
   but everyone else has full permissions.
 
 Note: the owner can always change a file's permissions.
 
 ---
 
 # Modifying permissions
 
 To modify a file's permissions, use .highlight[`chmod CHANGES FILE`]
 
 .mnemonic[
   `chmod` stands for .highlight[change mode].
 ]
 
 Possible changes:
 * .highlight[`+r`], .highlight[`+w`], .highlight[`+x` add] permissions for the owner.
   Can combine: `+rw`
 * .highlight[`-r` removes] permissions.
 * .highlight[`=r`] sets pemissions to .highlight[exactly] `r--`.
 * Prefix .highlight[`g`] changes permissions for the .highlight[group], e.g. `g+r`.
 * Prefix .highlight[`o`] changes permissions for .highlight[others], e.g. `o-w`.
 * Prefix .highlight[`a`] changes permissions for .highlight[all] three sets at once.
 * An .highlight[octal number] (e.g. 750) sets permissions to a specific configuration
 (in this case, `rwxr-x---`).
 
 ---
 
 # Modifying permissions
 
 Several changes can be applied at once, separated by commas:
 
 ```
 user@host:~/scits-training/io $ ls -la date.log
 -rwxrw-r-- 1 user group   90 Sep 10 23:08 date.log
 
 user@host:~/scits-training/io $ chmod +x,g=rx,o-r date.log
 
 user@host:~/scits-training/io $ ls -la date.log
 -rwxr-x--- 1 user group   90 Sep 10 23:08 date.log
 ```
 
 .exercise[
 Modify permissions on the file to be `r-xr--rw-`
 ]
 
 ---
 
 # Changing ownership
 
 Similarly to `chmod`, the .highlight[`chown`] command allows changing a file's owner and group.
 
 * `chown USER FILE` changes the owner
 * `chown :GROUP FILE` changes the group
 * `chown USER:GROUP FILE` changes both
 
 Note: once the owner is changed, the old owner no longer can modify access to the file.
 
 .highlight[For this reason, only administrators can change the file owner.]
 
 .exercise[
 Change a file's group to `root`, and then back to the original one.
 ]
 
 ---
 
 # Shell scripting
 
 Shell is not just an interface to launch other programs;
 it comes with its own scripting language to automate complex tasks.
 
 You can have variables, loops, conditionals – a full-featured programming language.
 
 We will only show the very basics.
 
 .exercise[
   Navigate to `~/scits-training/scripts` and open `boom.sh` in your favourite editor (nano, vim)
 ]
 
 ---
 
 layout: true
 
 # Shell scripting
 
 ```bash
 #!/bin/bash
 
 # I hope you get the reference
 echo "Someone set up us the bomb."
 for i in {5..1}
 do
   echo "$i.."
   sleep 1
 done
 explosion = "Boom!"
 echo $explosion
 ```
 
 ---
 
 The first line of the script is special:
 ```bash
 #!/bin/bash
 ```
 
 It's called a .highlight["shebang"] (for .highlight[shell] and "!" .highlight[bang]).
 
 It tells the shell what to execute the rest of the script with.
 Since we're writing a `bash` shell script, we put there the path to `/bin/bash` itself.
 
 ---
 
 Other lines starting with .highlight[`#`] are .highlight[comments]
 ```bash
 # I hope you get the reference
 ```
 
 They are ignored by `bash` and are used to leave notes to yourself or others.
 
 ---
 
 .highlight[`echo`] command outputs its arguments to STDIN.
 
 ```bash
 echo "Someone set up us the bomb."
 ```
 
 Quotes are used to make text with spaces in it a single argument; here, they are optional.
 
 ---
 
 .highlight[`for`] designates a .highlight[loop]: a .highlight[variable `i`] will change from 5 to 1.
 
 ```bash
 for i in {5..1}
 do
   # something
 done
 ```
 
 The code in `# something` will repeat with `i` as 5, 4, 3, 2 and 1.
 
 `do` and `done` delimit the bounds of the loop.
 
 ---
 
 One can use the variable in expressions .highlight[prefixed by `$`], i.e. .highlight[`$i`]:
 
 ```bash
 echo "$i.."
 ```
 
 If there is ambiguity as to where a variable name ends, use braces: .highlight[`${i}`].
 
 ---
 
 Variables can also be simply .highlight[assigned to]:
 
 ```bash
 explosion="Boom!"
 echo $explosion
 ```
 
 The lack of spaces around `=` is .highlight[significant].
 
 ---
 
 layout: false
 
 # Running a script
 
 OK, suppose we wrote the above script. How to execute it?
 
 1. We need to make sure that it's allowed to execute:
 
   ```
   user@host:~/scits-training/scripts $ chmod +x boom.sh
   ```
 
 2. For security reasons, the current directory is not automatically considered when
   starting other programs. We need to explicitly refer to it:
 
   ```
   user@host:~/scits-training/scripts $ ./boom.sh
   ```
 
 .exercise[
 1. Execute the script, saving its output to a file.
 
 2. Modify the script to count down from 10.
 ]
 
 ---
 
 # Scripting, take two
 
 The next script you will type out yourselves.
 
 Open `beer.sh` in your favourite editor.
 
 We'll write a simple script to determine if a user is old enough to drink beer.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 ```
 
 Any bash script should start with an appropriate shebang.
 
 We want to ask the user for his/her age; we can use the .highlight[`read`] command.
 
 ```bash
 # -n prevents a line break, and note the extra space
 echo -n "What's your age? "
 read age
 ```
 
 This will display a promt for the user and wait for input.
 The result is then stored in the variable `$age`.
 
 For simplicity, we will not check that the input is indeed a valid number.
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 ```
 
 We need to make a decision based on age; we need an if-then-else construct.
 
 ```bash
 if [$age -lt 16]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .highlight[`fi`] here is `if` reversed, to close the .highlight[`if`] statement.
 
 Conditionals in bash are a bit clunky, but .highlight[`-lt`] here stands for .highlight[less than].
 
 ---
 
 # Scripting, take two
 
 ```bash
 #!/bin/bash
 echo -n "What's your age? "
 read age
 if [$age -lt 16]
 then
   echo "You're too young to drink!"
 else
   echo "You're old enough, have a beer!"
 fi
 ```
 
 .exercise[
 1. Save this script to `beer.sh`.
 
 2. Change the file's mode to allow execution.
 
 3. Test the script with different values.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's add a little personal touch.
 
 `whoami` is a command that returns the username. Let's edit `beer.sh` to use it:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
 else
   echo "$(whoami), you're old enough, have a beer!"
 fi
 ```
 
 .highlight[`$(something)`] allows you to execute a command and substitute the result within another command.
 
 .exercise[
 Test the new additions.
 ]
 
 ---
 
 # Scripting improvements
 
 Let's read the age from the command line arguments.
 
 `bash` automatically populates .highlight[`$0`] with the .highlight[name of the executable],
 and .highlight[`$1`], .highlight[`$2`] and so on with .highlight[arguments].
 
 Let's use `$1` as age if it's defined:
 
 ```bash
 if [ $1 ]
 then
   age = $1
 else
   echo -n "What's your age? "
   read age
 fi
 ```
 
 .exercise[
 Test that `./beer.sh` now automatically get age from its first argument, and still asks if no argument is provided.
 ]
 
 ---
 
 # Return values
 
 Whenever a program terminates, it returns a single integer to the shell that called it;
   it's called the .highlight[return value].
 
 By convention:
 * .highlight[0] means "no error".
 * any .highlight[non-zero value] means "some kind of error".
 
 Let's return appropriate values:
 
 ```bash
 then
   echo "$(whoami), you're too young to drink!"
   exit 1
 else
   echo "$(whoami), you're old enough, have a beer!"
   exit 0
 fi
 ```
 
 ---
 
 # Chaining commands
 
 You can chain commands in shell with .highlight[`;`] or .highlight[`&&`].
 
 .highlight[`;`] will execute commands one by one, regardless of errors.
 
 ```
 $ command1; command2
 ```
 
 .highlight[`&&`] will only execute the next command only if the previous one returned 0, i.e. finished without errors.
 ```
 $ command1 && command2
 ```
 
 .exercise[
 1. Apply the return value changes to `beer.sh`
 
 2. Test it with `./beer.sh && echo "Cheers!"`
 ]
 
 ---
 
 # Customizing your environment
 
 If you interact with a system often, you may want to add various shortcuts and customizations
   to your shell.
 
+--
+
 Bash uses two files, `~/.bash_profile` and `~/.bashrc`, to allow you to apply such customizations
   on every login.
 
+Essentially, they are bash scripts that are run before the shell shows a prompt.
+
+--
+
 * `.bashrc` is applied when you open a new shell when already logged in.
 * `.bash_profile` is applied when you log in (e.g. though SSH).
 
-One can apply call `.bashrc` from `.bash_profile` to cover both cases.
+One can apply code in `.bashrc` from `.bash_profile` to cover both cases.
+
+---
+
+exclude: true
 
 ```bash
 # In .bash_profile
 # -f tests that file exists
 # source executes commands in the current shell
 if [ -f ~/.bashrc ]; then
   source ~/.bashrc
 fi
 ```
 
 ---
 
 # Environment variables
 
-Your profile files can set various .highlight[environment variables]
+Your profile files can set various .highlight[environment variables]:
+snippets of data inherited by programs running from shell.
+
+You can see your current environment variables with:
+
+```
+$ env | less
+```
+
+Some programs rely on environmental variables to change their behavior. Example:
+
+```bash
+# Will replace the default editor with vim in some commands
+export EDITOR=vim
+```
+
+This works exactly like setting a variable in a script, except for the extra
+  command .highlight[export], which propagates this variable to child processes.
+
+By convention, environment variables are UPPERCASE.
+
+---
+
+# Environment variables
+
+Your home directory is available in `$HOME` – in some contexts, using `~` is not possible.
+
+Another important variable is `$PATH`.
+
+It's a colon-separated list of directories which are searched when you try to run a program by name.
+
+Notably, the current directory is not in `$PATH`.
+
+If you have created some own scripts/programs and want them to be available by name from anywhere,
+  you can put them in a folder (e.g. `~/bin`) and add it to `$PATH`:
+
+```bash
+export PATH="$PATH:$HOME/bin"
+```
+
+---
+
+# Custom shell prompt
 
-TODO
+The variable `$PS1` contains the format template for your shell prompt.
+
+Throughout this training, we assumed the following:
+
+```bash
+export PS1="\u@\h:\w \\$ "
+```
+
+But you can customize it! Want the space for the command on a separate line?
+```bash
+export PS1="\u@\h:\w\\n\\$ " # \\n represents a line break
+```
+```
+user@host:~
+$ 
+```
+
+Want to add current time? Want to add some color?  
+There's [a guide](https://www.digitalocean.com/community/tutorials/how-to-customize-your-bash-prompt-on-a-linux-vps) for that.
+
+---
+
+# Aliases
+
+If you use a certain command often, you can define a short name for it.
+
+For example, if you want a shorter name for `ls -lh` because you always want to see human-readable sizes,
+you can make an alias:
+
+```
+$ alias lh="ls -lh"
+$ lh
+total 26M
+-rw-r--r-- 1 user group 25M Sep 11 07:22 big_file
+-rw-r--r-- 1 user group 735 Sep 11 07:22 description
+-rw-r--r-- 1 user group   0 Sep 11 07:22 empty_file
+-rw-r--r-- 1 user group 551 Sep 11 07:22 naming
+drwxr-xr-x 0 user group 512 Sep 11 07:22 subfolder
+```
+
+---
+
+# Making customizations permanent
+
+To make above tweaks permanent, they need to be added either to `.bash_profile` or `.bashrc`.
+
+Then they will apply on each opened shell.
+
+* `.bash_profile` is .highlight[sourced] at most once. Put things there that shouldn't be called multiple times.
+
+* `.bashrc` is sourced almost every time `bash` is called, except for initial SSH shell.
+  To be safe, you can "include" `.bashrc` into `.bash_profile` like this:
+
+```bash
+# In .bash_profile
+# -f tests that file exists
+# source executes commands in the current shell
+if [ -f ~/.bashrc ]; then
+  source ~/.bashrc
+fi
+```

\ No newline at end of file diff --git a/linux_2.zip b/linux_2.zip new file mode 100644 index 0000000..29d7c0f Binary files /dev/null and b/linux_2.zip differ