diff --git a/docs/configuration/overlay.rst b/docs/configuration/overlay.rst
index 3b31e2411..f1f93d60a 100644
--- a/docs/configuration/overlay.rst
+++ b/docs/configuration/overlay.rst
@@ -1,626 +1,631 @@
.. This file is part of Invenio
Copyright (C) 2014, 2015 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
.. _overlay:
==================================
How to create an Invenio overlay
==================================
+.. admonition:: CAVEAT LECTOR
+
+ The following overlay guide was written for Invenio v2.0 release series. It
+ is to be updated for Invenio v3.0.x
+
.. admonition:: TODO
The following items are still missing from this document, feel free to
contribute to it.
- celery/redis/honcho/flower
- populate the data
Thanks
What is an overlay
==================
Invenio is a library that enable the creation of an digital library but it
has to be coupled with an overlay. The overlay will contain your configuration
options, your desired look and feel, the extra invenio modules you've developed
and are using.
Creating your first Invenio overlay
===================================
If you've already setup the Invenio developer's environement itself, it will
feel familiar. We are reproducing here the steps for Ubuntu LTS. Other
distributions may be found in the previous link.
The global setup
----------------
Some softwares and libraries are required to work on your overlay. It's mostly
Python, MySQL, Redis as well as XML, XSLT and graphical libraries. Node.js is
solely required for development purposes.
.. code-block:: console
$ python --version
Python 2.7.5+
$ sudo apt-get update
$ sudo apt-get install build-essential git redis-server \
libmysqlclient-dev libxml2-dev libxslt-dev \
libjpeg-dev libfreetype6-dev libtiff-dev \
libffi-dev libssl-dev \
software-properties-common python-dev \
virtualenvwrapper
$ sudo pip install -U virtualenvwrapper pip
$ source .bashrc
# Install MySQL server, and keep the root password somewhere safe.
$ sudo apt-get install mysql-server
# Install node.js
$ curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
$ sudo apt-get install nodejs
The virtual environment
-----------------------
Python development usually recommends to work within a ``virtualenv``, which
creates an isolated environment where the libraries required by one will not
intervene with the system ones or the ones of another system. We are using
``virtualenvwrapper`` but nothing prevents your from directly using
``virtualenv`` you are familiar with it.
.. code-block:: console
$ mkvirtualenv myoverlay
(myoverlay)$ # we are in your overlay environment now and
(myoverlay)$ # can leave it using the deactivate command.
(myoverlay)$ deactivate
$ # Now join it back, recreating it would fail.
$ workon myoverlay
(myoverlay)$ # That's all there is to know about it.
The base of the overlay
-----------------------
Let's dive in.
.. code-block:: console
$ workon myoverlay
(myoverlay)$ cdvirtualenv
(myoverlay)$ mkdir -p src/myoverlay
(myoverlay)$ cd src/myoverlay
(myoverlay)$ edit setup.py
The ``setup.py`` file contains the definition of a python package. Having one
means you can rely on existing tools like ``pip`` to package, install, deploy
it later on. Here is its minimal content.
.. code-block:: python
from setuptools import setup
from setuptools import setup, find_packages
packages = find_packages()
setup(
name="My Overlay",
version="0.1.dev0",
url="http://invenio-software.org/",
author="Invenio Software",
author_email="invenio@invenio-software.org",
description="My first overlay",
packages=packages,
install_requires=[
"Invenio>=2"
],
entry_points={
"invenio.config": ["myoverlay = myoverlay.config"]
}
)
Now we can install it in editable mode (``-e``), meaning you don't have to
reinstall it after each change
.. code-block:: console
(myoverlay)$ pip install -e .
This will fetch the latest Invenio version published on PyPI. As a developer,
you may instead want to use the development version of Invenio from GitHub. To
do so, create a file called ``requirements.txt`` with the following content:
.. code-block:: text
git+git://github.com/inveniosoftware/invenio@pu#egg=Invenio-dev
-e .
and install using:
.. code-block:: console
(myoverlay)$ pip install -r requirements.txt
Configuration
=============
As you've seen above, we defined an entry_point for ``myoverlay.config``. It
points to a module that will contain our configuration. So create your
application.
.. code-block:: text
src/
│
├ myoverlay/
│ │
│ ├ base/
│ │ │
│ │ └ __init__.py
│ │
│ ├ __init__.py
│ └ config.py
│
├ requirements.txt
└ setup.py
Put the required configuration into ``config.py``.
.. code-block:: python
CFG_SITE_LANGS = ["en"]
CFG_SITE_NAME = "My Overlay"
CFG_SITE_NAME_INTL = {
"en": CFG_SITE_NAME
}
PACKAGES = [
"myoverlay.base",
# TODO list your packages here
"invenio_base",
]
try:
from myoverlay.instance_config import *
except ImportError:
pass
Sensitive configuration
-----------------------
Other configuration elements like database username and password or the website
url should not be put here as this file is not specific to the installation and
may be put under a version control system such as Git or Subversion.
The configuration can be handled via the `inveniomanage` command line interface
(or by editing the `invenio.cfg` file in the instance folder and reloading
the application).
.. code-block:: console
(myoverlay)$ inveniomanage config set create secret-key
# MySQL configuration
(myoverlay)$ inveniomanage config set CFG_DATABASE_NAME mysql-database
(myoverlay)$ inveniomanage config set CFG_DATABASE_USER mysql-user
# HOST configuration (for redirects, etc.)
(myoverlay)$ inveniomanage config set CFG_SITE_URL http://invenio.example.com
(myoverlay)$ inveniomanage config set CFG_SITE_SECURE_URL https://invenio.example.com
(myoverlay)$ inveniomanage config set DEBUG True
(myoverlay)$ inveniomanage config set ASSETS_DEBUG True
Database setup
--------------
.. code-block:: console
(invenio)$ inveniomanage database init --user=root --password=$MYSQL_ROOT --yes-i-know
...
>>> Database has been installed.
(invenio)$ inveniomanage database create
...
>>> Tables filled successfully.
Assets
------
Most of the JavaScript and CSS libraries used are not bundled with invenio
itself and needs to be downloaded via `bower `_. Bower is
configured using two files:
- `.bowerrc`: tells where the assets are downloaded
- `bower.json`: lists the dependencies to be downloaded
The ``bower.json`` can be automagically generated through the `inveniomanage`
command. However, we will have to tell `bower` the path we want the libraries
downloaded to. In most cases, it will be in the instance directory under the
`static/vendors` path.
.. code-block:: console
$ sudo su -c "npm install -g bower less clean-css requirejs uglify-js"
(myoverlay)$ cdvirtualenv var/invenio.base-instance
(myoverlay)$ echo '{"directory": "static/vendors"}' > .bowerrc
(myoverlay)$ inveniomanage bower > bower.json
(myoverlay)$ bower install
For invenio to see the static files from the ``myoverlay.base`` module, it
needs to declare a Flask blueprint. Create the following file:
``myoverlay/base/views.py``.
.. code-block:: python
from flask import Blueprint
blueprint = Blueprint(
"myoverlay",
__name__,
url_prefix="/",
template_folder="templates", # where your custom templates will go
static_folder="static" # where the assets go
)
The assets will now be collected into the instance static folder from your
overlay, invenio itself and every libraries it uses.
.. code-block:: console
(myoverlay)$ inveniomanage collect
Running
=======
.. code-block:: console
(myoverlay)$ inveniomanage runserver
Translations
============
Invenio comes with full internationalization and localization support
based on `Babel `_ library and `Flask-Babel
`_. All strings you want to
translate in your overlay have to be marked with ``_()``.
When you have all strings properly marked, it is time to prepare
catalog that contains all these strings for tranlations to desired
languages.
Configuration
-------------
First of all, you have to get into the source folder of your overlay and
create a configuration file for *Babel*.
.. code-block:: ini
[python: **.py]
encoding = utf-8
[jinja2: **/templates/**]
encoding = utf-8
extensions = jinja2.ext.autoescape.jinja2.ext.with_
Save it as ``babel.cfg`` next to your ``setup.py``. Before we run the
extraction tool we need to add section to configure translation directory
to ``setup.cfg``.
.. code-block:: ini
[compile_catalog]
directory = myoverlay/base/translations/
[extract_messages]
output-file = myoverlay/base/translations/myoverlay.pot
[init_catalog]
input-file = myoverlay/base/translations/myoverlay.pot
output-dir = myoverlay/base/translations/
[update_catalog]
input-file = myoverlay/base/translations/myoverlay.pot
output-dir = myoverlay/base/translations/
Message Extraction
------------------
Then it’s time to run the Babel string extraction with given
configuration:
.. code-block:: console
(myoverlay)$ python setup.py extract_messages
Create Catalog for New Language
-------------------------------
Once all translatable strings are extracted, one need to prepare catalogs
for new languages. Following example shows how to prepare new catalog for
French in PO (Portable Object) format.
.. code-block:: console
(myoverlay)$ python setup.py init_catalog -l fr
Now edit the ``myoverlay/base/translations/fr/LC_MESSAGES/messages.po``
file as needed.
Compiling Catalog
-----------------
Next step is to prepare MO (Machine Object) files in the format which is
defined by the GNU `gettext `_ tools
and the GNU `translation project
`_.
To compile the translations for use, pybabel integration with distutils
helps again:
.. code-block:: console
(myoverlay)$ python setup.py compile_catalog
If you install Invenio in development mode you must compile catalog also
from the Invenio directory project.
.. note::
You should tell git to ignore your compliled translation by running:
.. code-block:: console
$ echo \*.mo >> .gitignore
Updating Strings
----------------
It is pretty common that your strings in the code will change over the
time. Pybabel provides support for updating the translation catalog with
new strings or changing existing ones. What do you have to do? Create a
new ``myoverlay.pot`` like above and then let pybabel merge the changes:
.. code-block:: console
$ python setup.py update_catalog
Deployment
==========
Deploying Invenio is almost a piece of cake using `Fabric
`_. The following step are inspired by the Flask
documentation: `Deploying with Fabric
`_
Prerequisites
-------------
First, you need a server with remote access (SSH), where you've installed all
the python dependencies (e.g. ``build-essentials``, ``python-dev``,
``libmysqlclient-dev``, etc.).
Install `fabric` locally,
.. code-block:: console
$ pip install fabric
and create a boilerplate ``fabfile.py``:
.. code-block:: python
import json
from fabric.api import *
from fabric.utils import error
from fabric.contrib.files import exists
env.user = 'invenio' # remote username
env.directory = '/home/invenio/www' # remote directory
env.hosts = ['yourserver'] # list of servers
Preparing the tarball
---------------------
Before deploying anything, we need to locally prepare the python package to be
installed. Thanks to our ``setup.py`` file, it's very simple.
Beforehand, we have to generate the static assets into our static folder. By
doing so, it's not required to install anything related to node.js on your
server (no ``bower``, ``less``, ``uglifyjs``, etc.).
.. code-block:: python
@task
def pack():
"""Create a new source distribution as tarball."""
with open(".bowerrc") as fp:
bower = json.load(fp)
local("inveniomanage assets build --directory {directory}/gen"
.format(**bower))
return local("python setup.py sdist --formats=gztar", capture=False) \
.succeeded
Try it:
.. code-block:: console
$ fab pack
...
Done
$ ls dist/
My-Overlay-0.1.dev0.tar.gz
This is the package that will be installed on your server.
Creating the virtual environement
---------------------------------
We love virtual environments. We recommend you to install each version into its
own virtual env enabling quick rollbacks.
.. code-block:: python
@task
def create_virtualenv():
"""Create the virtualenv."""
package = local("python setup.py --fullname", capture=True).strip()
venv = "{0}/{1}".format(env.directory, package)
with cd(env.directory):
if exists(package):
return error("This version {0} is already installed."
.format(package))
return run("virtualenv {0}".format(package)).succeeded
Installing the package
----------------------
We can now upload the local tarball into the virtualenv, and install everything
there.
.. code-block:: python
@task
def install():
"""Install package."""
package = local("python setup.py --fullname", capture=True).strip()
venv = "{0}/{1}".format(env.directory, package)
if not exists(venv):
return error("Meh? I need a virtualenv first.")
# Upload the package and put it into our virtualenv.
put("dist/{0}.tar.gz".format(package), "/tmp/app.tgz")
run("mkdir -p {0}/src".format(venv))
with cd("{0}/src".format(venv)):
run("tar xzf /tmp/app.tgz")
run("rm -rf /tmp/app.tgz")
# Jump into the virtualenv and install stuff
with cd("{0}/src/{1}".format(venv, package)):
success = run("{0}/bin/python setup.py install".format(venv)
if success:
# post install
run("{0}/bin/inveniomanage collect".format(venv))
return success
Combining all the three steps:
.. code-block:: console
$ fab pack virtualenv install
Configuration
-------------
The setup doesn't have the ``invenio.cfg`` file that is generated via
``inveniomanage config``. You should do so manually.
Running the server
------------------
uWSGI is super simple and neat, all you need is two files. In the example
below, we've installed two versions of our overlay and a symbolic link is
pointing to the one we want to run.
.. code-block:: console
$ ls www/
current -> My-Overlay-0.1
My-Overlay-0.1.dev1
My-Overlay-0.1.dev2
My-Overlay-0.1
wsgi.py
uwsgi.ini
Let's create the ``wsgi.py`` file.
.. code-block:: python
from invenio_base.factory import create_wsgi_app
application = create_wsgi_app()
And the µWSGI configuration:
.. code-block:: python
[uwsgi]
http = 0.0.0.0:4000
master = true
processes = 4
die-on-term = true
vaccum = true
chdir = %d
virtualenv = %d/current/
module = wsgi:application
touch-reload = %d/wsgi.py
Let's run it.
.. code-block:: console
$ pip install uwsgi
$ uwsgi --ini uwsgi.ini
# or in daemon mode
$ uwsgi -d uwsgi.log --ini uwsgi.ini
If the new version causes troubles, going back to the old one is as fast as
changing the symbolic link and restarting the WSGI server.
.. code-block:: console
$ rm current
$ ln -s My-Overlay-0.1.dev1 current
$ touch wsgi.py
Dealing with versions
---------------------
One good idea is to use symlink to point to your current virtualenv and run
your overlay from there. Doing that via Fabric is left as an exercise to the
reader.
When installing a new version, copying the ``invenio.cfg`` file over is the
only requirements. Restarting the WSGI server is usually done by ``touch``-ing
the ``wsgi.py`` file.
diff --git a/docs/installation/installation-detailed.rst b/docs/installation/installation-detailed.rst
index bb9636739..46ede6d75 100644
--- a/docs/installation/installation-detailed.rst
+++ b/docs/installation/installation-detailed.rst
@@ -1,506 +1,506 @@
.. This file is part of Invenio
Copyright (C) 2014, 2015 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
.. _installation_detailed:
Detailed installation guide
===========================
.. admonition:: CAVEAT LECTOR
Invenio v3.0 alpha is a bleeding-edge developer preview version that is
scheduled for public release in Q1/2016.
Introduction
------------
In this installation guide, we'll create an Invenio digital library instance
using a multi-machine setup where separate services (such as the database server
and the web server) run on separate dedicated machines. Such a multi-machine
setup emulates to what one would typically use in production. (However, it is
very well possible to follow this guide and install all the services onto the
same "localhost", if one wants to.)
We'll use six dedicated machines running the following services:
============= ============= ====================
node IP runs
============= ============= ====================
web 192.168.50.10 Invenio web application
postgresql 192.168.50.11 `PostgreSQL `_ database server
redis 192.168.50.12 `Redis `_ caching service
elasticsearch 192.168.50.13 `Elasticsearch `_ information retrieval service
rabbitmq 192.168.50.14 `RabbitMQ `_ messaging service
worker 192.168.50.15 `Celery `_ worker node
============= ============= ====================
The instructions below are tested on Ubuntu 14.04 LTS (Trusty Tahr) and CentOS 7
operating systems. For other operating systems such as Mac OS X, you may want to
check out the "kickstart" set of scripts coming with the Invenio source code
that perform the below-quoted installation steps in an unattended automated way.
Environment variables
---------------------
Let's define some useful environment variables that will describe our Invenio
instance setup:
.. glossary::
INVENIO_WEB_HOST
The IP address of the Web server node.
INVENIO_WEB_INSTANCE
The name of your Invenio instance that will be created. Usually equal to
the name of the Python virtual environment.
INVENIO_WEB_VENV
The name of the Python virtual environment where Invenio will be installed.
Usually equal to the name of the Invenio instance.
INVENIO_USER_EMAIL
The email address of a user account that will be created on the Invenio
instance.
INVENIO_USER_PASS
The password of this Invenio user.
INVENIO_POSTGRESQL_HOST
The IP address of the PostgreSQL database server.
INVENIO_POSTGRESQL_DBNAME
The database name that will hold persistent data of our Invenio instance.
INVENIO_POSTGRESQL_DBUSER
The database user name used to connect to the database server.
INVENIO_POSTGRESQL_DBPASS
The password of this database user.
INVENIO_REDIS_HOST
The IP address af the Redis server.
INVENIO_ELASTICSEARCH_HOST
The IP address of the Elasticsearch information retrieval server.
INVENIO_RABBITMQ_HOST
The IP address of the RabbitMQ messaging server.
INVENIO_WORKER_HOST
The IP address of the Celery worker node.
In our example setup, we shall use:
.. include:: ../../.inveniorc
:start-after: # sphinxdoc-kickstart-configuration-variables-begin
:end-before: # sphinxdoc-kickstart-configuration-variables-end
:literal:
Let us save this configuration in a file called ``.inveniorc`` for future use.
Web
---
The web application node (192.168.50.10) is where the main Invenio application
will be running. We need to provision it with some system dependencies in order
to be able to install various underlying Python and JavaScript libraries.
The web application node can be set up in an automated unattended way by running
the following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-web.sh
Let's see in detail what the web provisioning script does.
First, some useful system tools are installed:
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-useful-system-tools-trusty-begin
:end-before: # sphinxdoc-install-useful-system-tools-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-useful-system-tools-centos7-begin
:end-before: # sphinxdoc-install-useful-system-tools-centos7-end
:literal:
Second, an external Node.js package repository is enabled. We'll be needing to
-install and run Bower on the web node later. The Node.js repository is enabled
+install and run Npm on the web node later. The Node.js repository is enabled
as follows:
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-add-nodejs-external-repository-trusty-begin
:end-before: # sphinxdoc-add-nodejs-external-repository-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-add-nodejs-external-repository-centos7-begin
:end-before: # sphinxdoc-add-nodejs-external-repository-centos7-end
:literal:
Third, all the common prerequisite software libraries and packages that Invenio
needs are installed:
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-web-common-trusty-begin
:end-before: # sphinxdoc-install-web-common-trusty-end
:literal:
* on CentOS7:
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-web-common-centos7-begin
:end-before: # sphinxdoc-install-web-common-centos7-end
:literal:
We want to use PostgreSQL database in this installation example, so we need to
install corresponding libraries too:
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-web-libpostgresql-trusty-begin
:end-before: # sphinxdoc-install-web-libpostgresql-trusty-end
:literal:
* on CentOS7:
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-web-libpostgresql-centos7-begin
:end-before: # sphinxdoc-install-web-libpostgresql-centos7-end
:literal:
-Fourth, now that Node.js is installed, we can proceed with installing Bower and
+Fourth, now that Node.js is installed, we can proceed with installing Npm and
associated CSS/JS filter tools. Let's do it globally:
* on either of the operating systems:
.. include:: ../../scripts/provision-web.sh
- :start-after: # sphinxdoc-install-bower-and-css-js-filters-begin
- :end-before: # sphinxdoc-install-bower-and-css-js-filters-end
+ :start-after: # sphinxdoc-install-npm-and-css-js-filters-begin
+ :end-before: # sphinxdoc-install-npm-and-css-js-filters-end
:literal:
Finally, we'll install Python virtual environment wrapper tools and activate
them in the current user shell process:
* on either of the operating systems:
.. include:: ../../scripts/provision-web.sh
:start-after: # sphinxdoc-install-virtualenvwrapper-begin
:end-before: # sphinxdoc-install-virtualenvwrapper-end
:literal:
Database
--------
The database server (192.168.50.11) will hold persistent data of our Invenio
installation, such as bibliographic records or user accounts. Invenio supports
MySQL, PostgreSQL, and SQLite databases. In this tutorial, we shall use
PostgreSQL that is the recommended database platform for Invenio.
The database server node can be set up in an automated unattended way by running
the following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-postgresql.sh
Let's see in detail what the database provisioning script does.
First, we install and configure the database software:
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-postgresql.sh
:start-after: # sphinxdoc-install-postgresql-trusty-begin
:end-before: # sphinxdoc-install-postgresql-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-postgresql.sh
:start-after: # sphinxdoc-install-postgresql-centos7-begin
:end-before: # sphinxdoc-install-postgresql-centos7-end
:literal:
We can now create a new database user with the necessary access permissions on
the new database:
* on either of the operating systems:
.. include:: ../../scripts/provision-postgresql.sh
:start-after: # sphinxdoc-setup-postgresql-access-begin
:end-before: # sphinxdoc-setup-postgresql-access-end
:literal:
Redis
-----
The Redis server (192.168.50.12) is used for various caching needs.
The Redis server can be set up in an automated unattended way by running the
following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-redis.sh
Let's see in detail what the Redis provisioning script does.
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-redis.sh
:start-after: # sphinxdoc-install-redis-trusty-begin
:end-before: # sphinxdoc-install-redis-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-redis.sh
:start-after: # sphinxdoc-install-redis-centos7-begin
:end-before: # sphinxdoc-install-redis-centos7-end
:literal:
Elasticsearch
-------------
The Elasticsearch server (192.168.50.13) is used to index and search
bibliographic records, fulltext documents, and other various interesting
information managed by our Invenio digital library instance.
The Elasticsearch server can be set up in an automated unattended way by running
the following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-elasticsearch.sh
Let's see in detail what the Elasticsearch provisioning script does.
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-elasticsearch.sh
:start-after: # sphinxdoc-install-elasticsearch-trusty-begin
:end-before: # sphinxdoc-install-elasticsearch-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-elasticsearch.sh
:start-after: # sphinxdoc-install-elasticsearch-centos7-begin
:end-before: # sphinxdoc-install-elasticsearch-centos7-end
:literal:
RabbitMQ
--------
The RabbitMQ server (192.168.50.14) is used as a messaging middleware broker.
The RabbitMQ server can be set up in an automated unattended way by running the
following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-rabbitmq.sh
Let's see in detail what the RabbitMQ provisioning script does.
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-rabbitmq.sh
:start-after: # sphinxdoc-install-rabbitmq-trusty-begin
:end-before: # sphinxdoc-install-rabbitmq-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-rabbitmq.sh
:start-after: # sphinxdoc-install-rabbitmq-centos7-begin
:end-before: # sphinxdoc-install-rabbitmq-centos7-end
:literal:
Worker
------
The Celery worker node (192.168.50.15) is used to execute potentially long tasks
in asynchronous manner.
The worker node can be set up in an automated unattended way by running the
following script:
.. code-block:: shell
source .inveniorc
./scripts/provision-worker.sh
Let's see in detail what the worker provisioning script does.
* on Ubuntu 14.04 LTS (Trusty Tahr):
.. include:: ../../scripts/provision-worker.sh
:start-after: # sphinxdoc-install-worker-trusty-begin
:end-before: # sphinxdoc-install-worker-trusty-end
:literal:
* on CentOS 7:
.. include:: ../../scripts/provision-worker.sh
:start-after: # sphinxdoc-install-worker-centos7-begin
:end-before: # sphinxdoc-install-worker-centos7-end
:literal:
Invenio
-------
Now that all the prerequisites have been set up, we can proceed with the
installation of the Invenio itself. The installation is happening on the web
node (192.168.50.10).
The installation of Invenio can be done in an automated unattended way by
running the following script:
.. code-block:: shell
source .inveniorc
./scripts/install.sh
Let's see in detail what the Invenio installation script does.
We start by creating a fresh new Python virtual environment that will hold our
Invenio v3.0 instance:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-create-virtual-environment-begin
:end-before: # sphinxdoc-create-virtual-environment-end
:literal:
We continue by installing Invenio v3.0 base package and most of its available
modules (by using option ``full`` as opposed to using option ``minimal``) from
PyPI:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-install-invenio-full-begin
:end-before: # sphinxdoc-install-invenio-full-end
:literal:
We can now create a new Invenio instance:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-create-instance-begin
:end-before: # sphinxdoc-create-instance-end
:literal:
Let's briefly customise our instance with respect to the location of the
database server, the Redis server, and all the other dependent services in our
multi-server environment:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-customise-instance-begin
:end-before: # sphinxdoc-customise-instance-end
:literal:
-In the instance folder, we run Bower to install any JavaScript libraries that
+In the instance folder, we run Npm to install any JavaScript libraries that
Invenio needs:
.. include:: ../../scripts/install.sh
- :start-after: # sphinxdoc-run-bower-begin
- :end-before: # sphinxdoc-run-bower-end
+ :start-after: # sphinxdoc-run-npm-begin
+ :end-before: # sphinxdoc-run-npm-end
:literal:
We can now collect and build CSS/JS assets of our Invenio instance:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-collect-and-build-assets-begin
:end-before: # sphinxdoc-collect-and-build-assets-end
:literal:
We proceed by creating a dedicated database that will hold persistent data of
our installation, such as bibliographic records or user accounts. The database
tables can be created as follows:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-create-database-begin
:end-before: # sphinxdoc-create-database-end
:literal:
We continue by creating a user account:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-create-user-account-begin
:end-before: # sphinxdoc-create-user-account-end
:literal:
Let's now start Celery worker that will execute instance tasks:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-start-celery-worker-begin
:end-before: # sphinxdoc-start-celery-worker-end
:literal:
Now that Celery is running, we can populate our Invenio instance with demo
records:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-populate-with-demo-records-begin
:end-before: # sphinxdoc-populate-with-demo-records-end
:literal:
Let's register persistent identifiers for the uploaded demo records:
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-register-pid-begin
:end-before: # sphinxdoc-register-pid-end
:literal:
Finally, let's start the web application (in debugging mode):
.. include:: ../../scripts/install.sh
:start-after: # sphinxdoc-start-application-begin
:end-before: # sphinxdoc-start-application-end
:literal:
We should now see our demo record on the web:
.. code-block:: shell
firefox http://localhost:5000/records/1
and we can access it via REST API:
.. code-block:: shell
curl -i -H "Accept: application/json" \
http://localhost:5000/api/records/1
We are done! Our first Invenio v3.0 instance is fully up and running.
diff --git a/docs/technology/docker.rst b/docs/technology/docker.rst
index 31d7c9a63..7ab6250b6 100644
--- a/docs/technology/docker.rst
+++ b/docs/technology/docker.rst
@@ -1,284 +1,284 @@
.. This file is part of Invenio
Copyright (C) 2015 CERN.
Invenio is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of the
License, or (at your option) any later version.
Invenio is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with Invenio; if not, write to the Free Software Foundation, Inc.,
59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
.. _developers-docker:
Docker
======
This page describes how to set up Docker containers for development purposes.
.. image:: /_static/docker-compose-setup.svg
Setup
-----
Install Docker_ and `Docker Compose`_. Now run:
.. code-block:: shell
docker-compose -f docker-compose-dev.yml build
docker-compose -f docker-compose-dev.yml up
This builds and runs the docker containers. You can now connect to
`localhost:28080` to see your Invenio installation. The `admin` user does not
have any password.
.. caution::
This will mirror the current source code directory into the Docker
container, so make sure to delete all `*.pyc`-files before. They might
not be compatible with the Python version and libraries of Docker image.
.. note::
If you are using `boot2docker`_ you need to set up port forwarding by
running the following command in a new terminal:
.. code-block:: shell
boot2docker ssh -vnNT \
-Llocalhost:29200:localhost:29200 \
-Llocalhost:28080:localhost:28080 \
-Llocalhost:26379:localhost:26379 \
-Llocalhost:25673:localhost:25673 \
-Llocalhost:25672:localhost:25672 \
-Llocalhost:23306:localhost:23306
You have to run this after Invenio booted up. Do **not** stop it while you
are working with Invenio. Otherwise the port forwarding gets stopped. You
have to restart the forwarding when your restart the Docker containers. The
process can be stopped using `CTRL-C`.
Should you require a fresh installation and therefore wipe all your instance
data, run:
.. code-block:: shell
docker-compose -f docker-compose-dev.yml rm -f
Interactive sessions
--------------------
For interactive sessions inside the container simply attach a shell to the
running instance:
.. code-block:: shell
docker exec -it invenio_web_1 bash
.. note::
Some tools (mostly ncurses-based ones) require the ability to detect the
used terminal. To enable this, set the `TERM` environment variable to the
correct value as a first command inside the container, e.g. by running:
.. code-block:: shell
export TERM=xterm
.. note::
Do not forget to run `bibsched` to schedule the processing of uploaded
records. You can put it into automatic mode if you like.
Debugging
---------
The `docker-compose-dev.yml` enables Werkzeug_, a debugger that
automatically kicks in whenever an error occurs. Stacktraces and
debugger terminal are available via web interface.
You can also insert a tracepoint into the code to start an interactive debugger
session:
.. code-block:: python
import ipdb; ipdb.set_tracepoint()
Furthermore you can debug MySQL at `localhost:23306`, Elasticsearch at
`localhost:29200`, RabbitMQ via `localhost:25672` (webinterface at
`localhost:25673`) and Redis at `localhost:26379`. You might want to use
flower_ for celery debugging and analysis as well. Just run the following
command to open the webinterface at port `5555`:
.. code-block:: shell
celery flower --broker=amqp://guest:guest@localhost:25672//
Should you require additional information about the behaviour of the different
containers as well as the contained processes and their interaction with the
system and other processes, the usage of classical Linux tools like Wireshark_
and sysdig_ might be helpful.
Code changes and live reloading
-------------------------------
.. note::
This section does not apply to OS X, Windows and boot2docker as these
systems are not properly supported by the used watchdog mechanism. When
you are using one of these setups, you have to restart the Docker
containers to reload the code and templates.
-As long as you do not add new requirements (python and bower) and only change
+As long as you do not add new requirements (python and npm) and only change
files inside the `invenio` package, it is not required to rebuild the docker
images. Code changes are mirrored to the containers. If Flask supports it, on
your system it will automatically reload the application when changes are
detected. This sometimes might lead to timeouts in your browser session. Do not
worry about this, but be aware to only save files when you are ready for
reloading.
As of this writing changing template files do not lead to application reloading
and do not purge caches. As a workaround you can simple alter one of the python
files, e.g. by using `touch`.
.. note::
Changing the Python source files will invalidate stored bytecode files. For
security reasons, these bytecode files can only be recreated by the root
user. This can be done via:
.. code-block:: shell
docker exec -it -u root invenio_web_1 python -O -m compileall .
Building documentation
----------------------
You can also use the Docker container to build the documentation. This can be
done by attaching to running container:
.. code-block:: shell
docker exec -it invenio_web_1 sphinx-build -nW docs docs/_build/html
.. note::
This needs do be done in a running or initialized container because it
requires that Invenio is set up correctly. Otherwise, the script will break
because of missing access rights.
Running tests
-------------
You can also run tests using the Docker containers. Wait until the containers
finished setup and the webservice is running. Then use:
.. code-block:: shell
docker exec -it invenio_web_1 python setup.py test
.. note:: Running the test requires the deactivation of redirection
debugging. You can achieve this by setting the configuration
variable `DEBUG_TB_INTERCEPT_REDIRECTS = False`. (Done for you by
default if you use ``docker-compose``.)
Overlays
--------
You might want to use build distribute overlays using Docker. Instead of
creating an entire new image and rewrite everything from scratch, you can the
Invenio Docker image. Start by building the image from a branch or release of
your choice:
.. code-block:: shell
cd src/invenio
docker build -t invenio:2.0 .
Now go to your overlay and create a Dockerfile that suits your needs, e.g:
.. code-block:: docker
# based on the right Invenio base image
FROM invenio:2.0
# get root rights again
USER root
# optional:
# add new packages
# (update apt caches, because it was cleaned from the base image)
# RUN apt-get update && \
# apt-get -qy install whatever_you_need
# optional:
# add new packages from pip
# RUN pip install what_suits_you
# optional:
# add new packages from npm
# RUN npm update && \
# npm install fun
# optional:
# make even more modifications
# add content
ADD . /code-overlay
WORKDIR /code-overlay
# fix requirements.txt and install additional dependencies
RUN sed -i '/inveniosoftware\/invenio[@#]/d' requirements.txt && \
pip install -r requirements.txt --exists-action i
# build
RUN python setup.py compile_catalog
# optional:
# do some cleanup
# step back
# in general code should not be writeable, especially because we are using
# `pip install -e`
RUN mkdir -p /code-overlay/src && \
chown -R invenio:invenio /code-overlay && \
chown -R root:root /code-overlay/invenio_demosite && \
chown -R root:root /code-overlay/scripts && \
chown -R root:root /code-overlay/setup.* && \
chown -R root:root /code-overlay/src
# finally step back again
USER invenio
Notice that this Dockerfile must be located in the directory of your
overlay. For a full working example, please see `invenio-demosite
`_.
Here is how to build the demo site:
.. code-block:: shell
cd ~/private/src/invenio
git checkout maint-2.0
docker build -t invenio:2.0 .
cd ~/private/src/invenio-demosite
git checkout maint-2.0
docker-compose -f docker-compose-dev.yml build
docker-compose -f docker-compose-dev.yml up
After all the daemons are up and running, you can populate the demo
site with demo records:
.. code-block:: shell
docker exec -i -t -u invenio inveniodemosite_web_1 \
inveniomanage demosite populate \
--packages=invenio_demosite.base --yes-i-know
Done. Your Invenio overlay installation is now up and running.
.. _boot2docker: http://boot2docker.io/
.. _Docker: https://www.docker.com/
.. _Docker Compose: https://docs.docker.com/compose/
.. _flower: https://flower.readthedocs.org/
.. _sysdig: http://www.sysdig.org/
.. _Werkzeug: http://werkzeug.pocoo.org/
.. _Wireshark: https://www.wireshark.org/
diff --git a/scripts/install.sh b/scripts/install.sh
index 894c244f1..cbeba70b9 100755
--- a/scripts/install.sh
+++ b/scripts/install.sh
@@ -1,177 +1,177 @@
#!/usr/bin/env bash
#
# This file is part of Invenio.
# Copyright (C) 2015 CERN.
#
# Invenio is free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of the
# License, or (at your option) any later version.
#
# Invenio is distributed in the hope that it will be
# useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Invenio; if not, write to the
# Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307, USA.
#
# In applying this license, CERN does not
# waive the privileges and immunities granted to it by virtue of its status
# as an Intergovernmental Organization or submit itself to any jurisdiction.
# check environment variables:
if [ -v ${INVENIO_WEB_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_WEB_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_WEB_HOST=192.168.50.10"
exit 1
fi
if [ -v ${INVENIO_WEB_INSTANCE} ]; then
echo "[ERROR] Please set environment variable INVENIO_WEB_INSTANCE before runnning this script."
echo "[ERROR] Example: export INVENIO_WEB_INSTANCE=invenio3"
exit 1
fi
if [ -v ${INVENIO_WEB_VENV} ]; then
echo "[ERROR] Please set environment variable INVENIO_WEB_VENV before runnning this script."
echo "[ERROR] Example: export INVENIO_WEB_VENV=invenio3"
exit 1
fi
if [ -v ${INVENIO_USER_EMAIL} ]; then
echo "[ERROR] Please set environment variable INVENIO_USER_EMAIL before runnning this script."
echo "[ERROR] Example: export INVENIO_USER_EMAIL=info@inveniosoftware.org"
exit 1
fi
if [ -v ${INVENIO_USER_PASS} ]; then
echo "[ERROR] Please set environment variable INVENIO_USER_PASS before runnning this script."
echo "[ERROR] Example: export INVENIO_USER_PASS=uspass123"
exit 1
fi
if [ -v ${INVENIO_POSTGRESQL_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_POSTGRESQL_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_POSTGRESQL_HOST=192.168.50.11"
exit 1
fi
if [ -v ${INVENIO_POSTGRESQL_DBNAME} ]; then
echo "[ERROR] Please set environment variable INVENIO_POSTGRESQL_DBNAME before runnning this script."
echo "[ERROR] Example: INVENIO_POSTGRESQL_DBNAME=invenio3"
exit 1
fi
if [ -v ${INVENIO_POSTGRESQL_DBUSER} ]; then
echo "[ERROR] Please set environment variable INVENIO_POSTGRESQL_DBUSER before runnning this script."
echo "[ERROR] Example: INVENIO_POSTGRESQL_DBUSER=invenio3"
exit 1
fi
if [ -v ${INVENIO_POSTGRESQL_DBPASS} ]; then
echo "[ERROR] Please set environment variable INVENIO_POSTGRESQL_DBPASS before runnning this script."
echo "[ERROR] Example: INVENIO_POSTGRESQL_DBPASS=dbpass123"
exit 1
fi
if [ -v ${INVENIO_REDIS_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_REDIS_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_REDIS_HOST=192.168.50.12"
exit 1
fi
if [ -v ${INVENIO_ELASTICSEARCH_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_ELASTICSEARCH_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_ELASTICSEARCH_HOST=192.168.50.13"
exit 1
fi
if [ -v ${INVENIO_RABBITMQ_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_RABBITMQ_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_RABBITMQ_HOST=192.168.50.14"
exit 1
fi
if [ -v ${INVENIO_WORKER_HOST} ]; then
echo "[ERROR] Please set environment variable INVENIO_WORKER_HOST before runnning this script."
echo "[ERROR] Example: export INVENIO_WORKER_HOST=192.168.50.15"
exit 1
fi
# load virtualenvrapper:
source $(which virtualenvwrapper.sh)
# sphinxdoc-create-virtual-environment-begin
mkvirtualenv ${INVENIO_WEB_VENV}
cdvirtualenv
mkdir -p src
cd src
# sphinxdoc-create-virtual-environment-end
# quit on errors and unbound symbols:
set -o errexit
set -o nounset
# sphinxdoc-install-invenio-full-begin
# FIXME we pin older version that works:
pip install invenio-records==v1.0.0a3
# FIXME we need to install PostgreSQL DB connect option explicitly:
pip install invenio-db[postgresql]
# now we can install full Invenio:
pip install invenio[full] --pre
# sphinxdoc-install-invenio-full-end
# sphinxdoc-create-instance-begin
inveniomanage instance create ${INVENIO_WEB_INSTANCE}
# sphinxdoc-create-instance-end
# sphinxdoc-customise-instance-begin
mkdir -p ../var/${INVENIO_WEB_INSTANCE}-instance/
echo "# Database" > ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "SQLALCHEMY_DATABASE_URI='postgresql+psycopg2://${INVENIO_POSTGRESQL_DBUSER}:${INVENIO_POSTGRESQL_DBPASS}@${INVENIO_POSTGRESQL_HOST}:5432/${INVENIO_POSTGRESQL_DBNAME}'" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "# Redis" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "CACHE_REDIS_HOST='${INVENIO_REDIS_HOST}'" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "BROKER_URL='redis://${INVENIO_REDIS_HOST}:6379/0'" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
echo "CELERY_RESULT_BACKEND='redis://${INVENIO_REDIS_HOST}:6379/1'" >> ../var/${INVENIO_WEB_INSTANCE}-instance/${INVENIO_WEB_INSTANCE}.cfg
# sphinxdoc-customise-instance-end
-# sphinxdoc-run-bower-begin
+# sphinxdoc-run-npm-begin
cd ${INVENIO_WEB_INSTANCE}
pip install ipaddr # FIXME this seems explicitly needed on Python-2.7
-python manage.py bower
-cdvirtualenv var/${INVENIO_WEB_INSTANCE}-instance/
-CI=true bower install
+python manage.py npm
+cdvirtualenv var/${INVENIO_WEB_INSTANCE}-instance/static
+CI=true npm install
cd -
-# sphinxdoc-run-bower-end
+# sphinxdoc-run-npm-end
# sphinxdoc-collect-and-build-assets-begin
python manage.py collect -v
python manage.py assets build
# sphinxdoc-collect-and-build-assets-end
# sphinxdoc-create-database-begin
# python manage.py db init # FIXME our database is initialised already
python manage.py db create
# sphinxdoc-create-database-end
# sphinxdoc-create-user-account-begin
python manage.py users create \
--email ${INVENIO_USER_EMAIL} \
--password ${INVENIO_USER_PASS} \
--active
# sphinxdoc-create-user-account-end
# sphinxdoc-start-celery-worker-begin
# FIXME we should run celery worker on another node
celery worker -A ${INVENIO_WEB_INSTANCE}.celery -l INFO &
# sphinxdoc-start-celery-worker-end
# sphinxdoc-populate-with-demo-records-begin
echo '{"title":"Invenio 3 Rocks", "recid": 1}'| \
python manage.py records create
# sphinxdoc-populate-with-demo-records-end
# sphinxdoc-register-pid-begin
echo "from invenio_db import db; \
from invenio_pidstore.models import PersistentIdentifier; \
pid = PersistentIdentifier.create('recid', '1', 'recid'); \
pid.assign('rec', '1'); \
pid.register(); \
db.session.commit()" | python manage.py shell
# sphinxdoc-register-pid-end
# sphinxdoc-start-application-begin
python manage.py --debug run &
# sphinxdoc-start-application-end
diff --git a/scripts/provision-web.sh b/scripts/provision-web.sh
index a53598ce9..e6d463f65 100755
--- a/scripts/provision-web.sh
+++ b/scripts/provision-web.sh
@@ -1,184 +1,184 @@
#!/usr/bin/env bash
#
# This file is part of Invenio.
# Copyright (C) 2015 CERN.
#
# Invenio is free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of the
# License, or (at your option) any later version.
#
# Invenio is distributed in the hope that it will be
# useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Invenio; if not, write to the
# Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307, USA.
#
# In applying this license, CERN does not
# waive the privileges and immunities granted to it by virtue of its status
# as an Intergovernmental Organization or submit itself to any jurisdiction.
# quit on errors:
set -o errexit
# quit on unbound symbols:
set -o nounset
provision_web_common_ubuntu_trusty () {
# sphinxdoc-install-useful-system-tools-trusty-begin
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install \
curl \
git \
rlwrap \
screen \
vim
# sphinxdoc-install-useful-system-tools-trusty-end
# sphinxdoc-add-nodejs-external-repository-trusty-begin
if [[ ! -f /etc/apt/sources.list.d/nodesource.list ]]; then
curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
fi
# sphinxdoc-add-nodejs-external-repository-trusty-end
# sphinxdoc-install-web-common-trusty-begin
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install \
libffi-dev \
libfreetype6-dev \
libjpeg-dev \
libmsgpack-dev \
libssl-dev \
libtiff-dev \
libxml2-dev \
libxslt-dev \
nodejs \
python-dev \
python-pip
# sphinxdoc-install-web-common-trusty-end
}
provision_web_libpostgresql_ubuntu_trusty () {
# sphinxdoc-install-web-libpostgresql-trusty-begin
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install \
libpq-dev
# sphinxdoc-install-web-libpostgresql-trusty-end
}
provision_web_common_centos7 () {
# sphinxdoc-install-useful-system-tools-centos7-begin
sudo yum install -y \
curl \
git \
rlwrap \
screen \
vim
# sphinxdoc-install-useful-system-tools-centos7-end
# sphinxdoc-add-nodejs-external-repository-centos7-begin
# add EPEL external repository:
sudo yum install -y epel-release
# sphinxdoc-add-nodejs-external-repository-centos7-end
# sphinxdoc-install-web-common-centos7-begin
sudo yum update -y
sudo yum groupinstall -y "Development Tools"
sudo yum install -y \
libffi-devel \
libxml2-devel \
libxslt-devel \
npm \
python-devel \
python-pip
# sphinxdoc-install-web-common-centos7-end
}
provision_web_libpostgresql_centos7 () {
# sphinxdoc-install-web-libpostgresql-centos7-begin
sudo yum install -y \
postgresql-devel
# sphinxdoc-install-web-libpostgresql-centos7-end
}
-setup_bower_and_css_js_filters () {
+setup_npm_and_css_js_filters () {
- # sphinxdoc-install-bower-and-css-js-filters-begin
- sudo su -c "npm install -g bower"
+ # sphinxdoc-install-npm-and-css-js-filters-begin
+ sudo su -c "npm install -g npm"
sudo su -c "npm install -g node-sass clean-css requirejs uglify-js"
- # sphinxdoc-install-bower-and-css-js-filters-end
+ # sphinxdoc-install-npm-and-css-js-filters-end
}
setup_virtualenvwrapper () {
# disable quitting on errors due to virtualenvrapper:
set +o errexit
set +o nounset
# sphinxdoc-install-virtualenvwrapper-begin
sudo pip install -U virtualenvwrapper pip
if ! grep -q virtualenvwrapper ~/.bashrc; then
mkdir -p $HOME/.virtualenvs
echo "export WORKON_HOME=$HOME/.virtualenvs" >> $HOME/.bashrc
echo "source $(which virtualenvwrapper.sh)" >> $HOME/.bashrc
fi
export WORKON_HOME=$HOME/.virtualenvs
source $(which virtualenvwrapper.sh)
# sphinxdoc-install-virtualenvwrapper-end
# enable quitting on errors back:
set -o errexit
set -o nounset
}
main () {
# detect OS distribution and release version:
if hash lsb_release 2> /dev/null; then
os_distribution=$(lsb_release -i | cut -f 2)
os_release=$(lsb_release -r | cut -f 2)
elif [ -e /etc/redhat-release ]; then
os_distribution=$(cat /etc/redhat-release | cut -d ' ' -f 1)
os_release=$(cat /etc/redhat-release | grep -oE '[0-9]+\.' | cut -d. -f1 | head -1)
else
os_distribution="UNDETECTED"
os_release="UNDETECTED"
fi
# call appropriate provisioning functions:
if [ "$os_distribution" = "Ubuntu" ]; then
if [ "$os_release" = "14.04" ]; then
provision_web_common_ubuntu_trusty
provision_web_libpostgresql_ubuntu_trusty
else
echo "[ERROR] Sorry, unsupported release ${os_release}."
exit 1
fi
elif [ "$os_distribution" = "CentOS" ]; then
if [ "$os_release" = "7" ]; then
provision_web_common_centos7
provision_web_libpostgresql_centos7
else
echo "[ERROR] Sorry, unsupported release ${os_release}."
exit 1
fi
else
echo "[ERROR] Sorry, unsupported distribution ${os_distribution}."
exit 1
fi
# finish with common setups:
- setup_bower_and_css_js_filters
+ setup_npm_and_css_js_filters
setup_virtualenvwrapper
}
main