Homec4science
Diffusion invenio-infoscience (iugw2017) Browse

R3600/docs/technology/docker.rst
122e918bfb1biugw2017

Last Change
Raw File

docker.rst
View Options

.. This file is part of Invenio
Copyright (C) 2015 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
.. _developers-docker:
Docker
======
This page describes how to set up Docker containers for development purposes.
.. image:: /_static/docker-compose-setup.svg
Setup
-----
Install Docker_ and `Docker Compose`_. Now run:
.. code-block:: shell
docker-compose -f docker-compose-dev.yml build
docker-compose -f docker-compose-dev.yml up
This builds and runs the docker containers. You can now connect to
`localhost:28080` to see your Invenio installation. The `admin` user does not
have any password.
.. caution::
This will mirror the current source code directory into the Docker
container, so make sure to delete all `*.pyc`-files before. They might
not be compatible with the Python version and libraries of Docker image.
.. note::
If you are using `boot2docker`_ you need to set up port forwarding by
running the following command in a new terminal:
.. code-block:: shell
boot2docker ssh -vnNT \
-Llocalhost:29200:localhost:29200 \
-Llocalhost:28080:localhost:28080 \
-Llocalhost:26379:localhost:26379 \
-Llocalhost:25673:localhost:25673 \
-Llocalhost:25672:localhost:25672 \
-Llocalhost:23306:localhost:23306
You have to run this after Invenio booted up. Do **not** stop it while you
are working with Invenio. Otherwise the port forwarding gets stopped. You
have to restart the forwarding when your restart the Docker containers. The
process can be stopped using `CTRL-C`.
Should you require a fresh installation and therefore wipe all your instance
data, run:
.. code-block:: shell
docker-compose -f docker-compose-dev.yml rm -f
Interactive sessions
--------------------
For interactive sessions inside the container simply attach a shell to the
running instance:
.. code-block:: shell
docker exec -it invenio_web_1 bash
.. note::
Some tools (mostly ncurses-based ones) require the ability to detect the
used terminal. To enable this, set the `TERM` environment variable to the
correct value as a first command inside the container, e.g. by running:
.. code-block:: shell
export TERM=xterm
.. note::
Do not forget to run `bibsched` to schedule the processing of uploaded
records. You can put it into automatic mode if you like.
Debugging
---------
The `docker-compose-dev.yml` enables Werkzeug_, a debugger that
automatically kicks in whenever an error occurs. Stacktraces and
debugger terminal are available via web interface.
You can also insert a tracepoint into the code to start an interactive debugger
session:
.. code-block:: python
import ipdb; ipdb.set_tracepoint()
Furthermore you can debug MySQL at `localhost:23306`, Elasticsearch at
`localhost:29200`, RabbitMQ via `localhost:25672` (webinterface at
`localhost:25673`) and Redis at `localhost:26379`. You might want to use
flower_ for celery debugging and analysis as well. Just run the following
command to open the webinterface at port `5555`:
.. code-block:: shell
celery flower --broker=amqp://guest:guest@localhost:25672//
Should you require additional information about the behaviour of the different
containers as well as the contained processes and their interaction with the
system and other processes, the usage of classical Linux tools like Wireshark_
and sysdig_ might be helpful.
Code changes and live reloading
-------------------------------
.. note::
This section does not apply to OS X, Windows and boot2docker as these
systems are not properly supported by the used watchdog mechanism. When
you are using one of these setups, you have to restart the Docker
containers to reload the code and templates.
As long as you do not add new requirements (python and bower) and only change
files inside the `invenio` package, it is not required to rebuild the docker
images. Code changes are mirrored to the containers. If Flask supports it, on
your system it will automatically reload the application when changes are
detected. This sometimes might lead to timeouts in your browser session. Do not
worry about this, but be aware to only save files when you are ready for
reloading.
As of this writing changing template files do not lead to application reloading
and do not purge caches. As a workaround you can simple alter one of the python
files, e.g. by using `touch`.
.. note::
Changing the Python source files will invalidate stored bytecode files. For
security reasons, these bytecode files can only be recreated by the root
user. This can be done via:
.. code-block:: shell
docker exec -it -u root invenio_web_1 python -O -m compileall .
Building documentation
----------------------
You can also use the Docker container to build the documentation. This can be
done by attaching to running container:
.. code-block:: shell
docker exec -it invenio_web_1 sphinx-build -nW docs docs/_build/html
.. note::
This needs do be done in a running or initialized container because it
requires that Invenio is set up correctly. Otherwise, the script will break
because of missing access rights.
Running tests
-------------
You can also run tests using the Docker containers. Wait until the containers
finished setup and the webservice is running. Then use:
.. code-block:: shell
docker exec -it invenio_web_1 python setup.py test
.. note:: Running the test requires the deactivation of redirection
debugging. You can achieve this by setting the configuration
variable `DEBUG_TB_INTERCEPT_REDIRECTS = False`. (Done for you by
default if you use ``docker-compose``.)
Overlays
--------
You might want to use build distribute overlays using Docker. Instead of
creating an entire new image and rewrite everything from scratch, you can the
Invenio Docker image. Start by building the image from a branch or release of
your choice:
.. code-block:: shell
cd src/invenio
docker build -t invenio:2.0 .
Now go to your overlay and create a Dockerfile that suits your needs, e.g:
.. code-block:: docker
# based on the right Invenio base image
FROM invenio:2.0
# get root rights again
USER root
# optional:
# add new packages
# (update apt caches, because it was cleaned from the base image)
# RUN apt-get update && \
# apt-get -qy install whatever_you_need
# optional:
# add new packages from pip
# RUN pip install what_suits_you
# optional:
# add new packages from npm
# RUN npm update && \
# npm install fun
# optional:
# make even more modifications
# add content
ADD . /code-overlay
WORKDIR /code-overlay
# fix requirements.txt and install additional dependencies
RUN sed -i '/inveniosoftware\/invenio[@#]/d' requirements.txt && \
pip install -r requirements.txt --exists-action i
# build
RUN python setup.py compile_catalog
# optional:
# do some cleanup
# step back
# in general code should not be writeable, especially because we are using
# `pip install -e`
RUN mkdir -p /code-overlay/src && \
chown -R invenio:invenio /code-overlay && \
chown -R root:root /code-overlay/invenio_demosite && \
chown -R root:root /code-overlay/scripts && \
chown -R root:root /code-overlay/setup.* && \
chown -R root:root /code-overlay/src
# finally step back again
USER invenio
Notice that this Dockerfile must be located in the directory of your
overlay. For a full working example, please see `invenio-demosite
<https://github.com/inveniosoftware/invenio-demosite/tree/maint-2.0>`_.
Here is how to build the demo site:
.. code-block:: shell
cd ~/private/src/invenio
git checkout maint-2.0
docker build -t invenio:2.0 .
cd ~/private/src/invenio-demosite
git checkout maint-2.0
docker-compose -f docker-compose-dev.yml build
docker-compose -f docker-compose-dev.yml up
After all the daemons are up and running, you can populate the demo
site with demo records:
.. code-block:: shell
docker exec -i -t -u invenio inveniodemosite_web_1 \
inveniomanage demosite populate \
--packages=invenio_demosite.base --yes-i-know
Done. Your Invenio overlay installation is now up and running.
.. _boot2docker: http://boot2docker.io/
.. _Docker: https://www.docker.com/
.. _Docker Compose: https://docs.docker.com/compose/
.. _flower: https://flower.readthedocs.org/
.. _sysdig: http://www.sysdig.org/
.. _Werkzeug: http://werkzeug.pocoo.org/
.. _Wireshark: https://www.wireshark.org/
c4science · Help