Homec4science
Diffusion invenio-infoscience (maint-1.0) Browse

R3600/INSTALL
d9cdbfc9045fmaint-1.0

Last Change
Raw File

INSTALL
View Options

CDS Invenio v0.99.1 INSTALL
===========================
About
=====
This document specifies how to build, customize, and install CDS
Invenio v0.99.1 for the first time. See RELEASE-NOTES if you are
upgrading from a previous CDS Invenio release.
Contents
========
0. Prerequisites
1. Quick instructions for the impatient CDS Invenio admin
2. Detailed instructions for the patient CDS Invenio admin
0. Prerequisites
================
Here is the software you need to have around before you
start installing CDS Invenio:
a) Unix-like operating system. The main development and
production platforms for CDS Invenio at CERN are GNU/Linux
distributions Debian, Gentoo, Scientific Linux (aka RHEL),
Ubuntu, but we also develop on Mac OS X. Basically any Unix
system supporting the software listed below should do.
If you are using Debian GNU/Linux ``Lenny'' or later, then you
can install most of the below-mentioned prerequisites and
recommendations by running:
$ sudo aptitude install python-dev apache2-mpm-prefork \
mysql-server mysql-client python-mysqldb \
python-4suite-xml python-simplejson python-xml \
python-libxml2 python-libxslt1 gnuplot poppler-utils \
gs-common clisp gettext libapache2-mod-wsgi unzip \
python-dateutil python-rdflib \
python-gnuplot python-magic pdftk html2text giflib-tools \
pstotext netpbm
You may also want to install some of the following packages,
if you have them available on your concrete architecture:
$ sudo aptitude install rxp python-psyco sbcl cmucl \
pylint pychecker pyflakes python-profiler python-epydoc \
libapache2-mod-xsendfile openoffice.org
Moreover, you should install some Message Transfer Agent (MTA)
such as Postfix so that CDS Invenio can email notification
alerts or registration information to the end users, contact
moderators and reviewers of submitted documents, inform
administrators about various runtime system information, etc:
$ sudo aptitude install postfix
After running the above-quoted aptitude command(s), you can
proceed to configuring your MySQL server instance
(max_allowed_packet in my.cnf, see item 0b below) and then to
installing the CDS Invenio software package in the section 1
below.
If you are using another operating system, then please
continue reading the rest of this prerequisites section, and
please consult our wiki pages for any concrete hints for your
specific operating system.
<https://twiki.cern.ch/twiki/bin/view/CDS/Invenio>
b) MySQL server (may be on a remote machine), and MySQL client
(must be available locally too). MySQL versions 4.1 or 5.0
are supported. Please set the variable "max_allowed_packet"
in your "my.cnf" init file to at least 4M. You may perhaps
also want to run your MySQL server natively in UTF-8 mode by
setting "default-character-set=utf8" in various parts of your
"my.cnf" file, such as in the "[mysql]" part and elsewhere;
but this is not really required.
<http://mysql.com/>
c) Apache 2 server, with support for loading DSO modules, and
optionally with SSL support for HTTPS-secure user
authentication, and mod_xsendfile for off-loading file
downloads away from Invenio processes to Apache.
<http://httpd.apache.org/>
<http://tn123.ath.cx/mod_xsendfile/>
d) Python v2.4 or above:
<http://python.org/>
as well as the following Python modules:
- (mandatory) MySQLdb (version >= 1.2.1_p2; see below)
<http://sourceforge.net/projects/mysql-python>
- (recommended) python-dateutil, for complex date processing:
<http://labix.org/python-dateutil>
- (recommended) PyXML, for XML processing:
<http://pyxml.sourceforge.net/topics/download.html>
- (recommended) PyRXP, for very fast XML MARC processing:
<http://www.reportlab.org/pyrxp.html>
- (recommended) libxml2-python, for XML/XLST processing:
<ftp://xmlsoft.org/libxml2/python/>
- (recommended) simplejson, for AJAX apps:
<http://undefined.org/python/#simplejson>
Note that if you are using Python-2.6, you don't need to
install simplejson, because the module is already included
in the main Python distribution.
- (recommended) Gnuplot.Py, for producing graphs:
<http://gnuplot-py.sourceforge.net/>
- (recommended) Snowball Stemmer, for stemming:
<http://snowball.tartarus.org/wrappers/PyStemmer-1.0.1.tar.gz>
- (recommended) py-editdist, for record merging:
<http://www.mindrot.org/projects/py-editdist/>
- (recommended) numpy, for citerank methods:
<http://numpy.scipy.org/>
- (recommended) magic, for full-text file handling:
<http://www.darwinsys.com/file/>
- (optional) 4suite, slower alternative to PyRXP and
libxml2-python:
<http://4suite.org/>
- (optional) feedparser, for web journal creation:
<http://feedparser.org/>
- (optional) Psyco, if you are running on a 32-bit OS:
<http://psyco.sourceforge.net/>
- (optional) RDFLib, to use RDF ontologies and thesauri:
<http://rdflib.net/>
- (optional) mechanize, to run regression web test suite:
<http://wwwsearch.sourceforge.net/mechanize/>
Note: MySQLdb version 1.2.1_p2 or higher is recommended. If
you are using an older version of MySQLdb, you may get
into problems with character encoding.
e) mod_wsgi Apache module.
<http://code.google.com/p/modwsgi/>
Note: for the time being, the WSGI daemon must be run with
threads=1, because Invenio is not fully thread safe yet.
This will come later. The Apache configuration example
snippets (created below) will use threads=1.
Note: if you are using Python 2.4 or earlier, then you should
also install the wsgiref Python module, available from:
<http://pypi.python.org/pypi/wsgiref/> (As of Python 2.5
this module is included in standard Python
distribution.)
f) If you want to be able to extract references from PDF fulltext
files, then you need to install pdftotext version 3 at least.
<http://poppler.freedesktop.org/>
<http://www.foolabs.com/xpdf/home.html>
g) If you want to be able to search for words in the fulltext
files (i.e. to have fulltext indexing) or to stamp submitted
files, then you need as well to install some of the following
tools:
- for Microsoft Office/OpenOffice.org document conversion:
OpenOffice.org
<http://www.openoffice.org/>
- for PDF file stamping: pdftk, pdf2ps
<http://www.accesspdf.com/pdftk/>
<http://www.cs.wisc.edu/~ghost/doc/AFPL/>
- for PDF files: pdftotext or pstotext
<http://poppler.freedesktop.org/>
<http://www.foolabs.com/xpdf/home.html>
<http://www.cs.wisc.edu/~ghost/doc/AFPL/>
- for PostScript files: pstotext or ps2ascii
<http://www.cs.wisc.edu/~ghost/doc/AFPL/>
- for DjVu creation, elaboration: DjVuLibre
<http://djvu.sourceforge.net>
- to perform OCR: OCRopus (tested only with release 0.3.1)
<http://code.google.com/p/ocropus/>
- to perform different image elaborations: ImageMagick
<http://www.imagemagick.org/>
- to generate PDF after OCR: ReportLab
<http://www.reportlab.org/rl_toolkit.html>
- to analyze images to generate PDF after OCR: netpbm
<http://netpbm.sourceforge.net/>
h) If you have chosen to install fast XML MARC Python processors
in the step d) above, then you have to install the parsers
themselves:
- (optional) 4suite:
<http://4suite.org/>
i) (recommended) Gnuplot, the command-line driven interactive
plotting program. It is used to display download and citation
history graphs on the Detailed record pages on the web
interface. Note that Gnuplot must be compiled with PNG output
support, that is, with the GD library. Note also that Gnuplot
is not required, only recommended.
<http://www.gnuplot.info/>
j) (recommended) A Common Lisp implementation, such as CLISP,
SBCL or CMUCL. It is used for the web server log analysing
tool and the metadata checking program. Note that any of the
three implementations CLISP, SBCL, or CMUCL will do. CMUCL
produces fastest machine code, but it does not support UTF-8
yet. Pick up CLISP if you don't know what to do. Note that a
Common Lisp implementation is not required, only recommended.
<http://clisp.cons.org/>
<http://www.cons.org/cmucl/>
<http://sbcl.sourceforge.net/>
k) GNU gettext, a set of tools that makes it possible to
translate the application in multiple languages.
<http://www.gnu.org/software/gettext/>
This is available by default on many systems.
Note that the configure script checks whether you have all the
prerequisite software installed and that it won't let you continue
unless everything is in order. It also warns you if it cannot find
some optional but recommended software.
1. Quick instructions for the impatient CDS Invenio admin
=========================================================
1a. Installation
----------------
$ cd $HOME/src/
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.md5
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.sig
$ md5sum -v -c cds-invenio-0.99.1.tar.gz.md5
$ gpg --verify cds-invenio-0.99.1.tar.gz.sig cds-invenio-0.99.1.tar.gz
$ tar xvfz cds-invenio-0.99.1.tar.gz
$ cd cds-invenio-0.99.1
$ ./configure
$ make
$ make install
$ make install-jsmath-plugin ## optional
$ make install-jquery-plugins ## optional
$ make install-fckeditor-plugin ## optional
$ make install-pdfa-helper-files ## optional
1b. Configuration
-----------------
$ sudo chown -R www-data.www-data /opt/cds-invenio
$ sudo -u www-data emacs /opt/cds-invenio/etc/invenio-local.conf
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --update-all
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-tables
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-webstat-conf
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-apache-conf
$ sudo /etc/init.d/apache2 restart
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-demo-site
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-demo-records
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-unit-tests
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-regression-tests
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-web-tests
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --remove-demo-records
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --drop-demo-site
$ firefox http://your.site.com/help/admin/howto-run
2. Detailed instructions for the patient CDS Invenio admin
==========================================================
2a. Installation
----------------
The CDS Invenio uses standard GNU autoconf method to build and
install its files. This means that you proceed as follows:
$ cd $HOME/src/
Change to a directory where we will build the CDS Invenio
sources. (The built files will be installed into different
"target" directories later.)
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.md5
$ wget http://cdsware.cern.ch/download/cds-invenio-0.99.1.tar.gz.sig
Fetch CDS Invenio source tarball from the CDS Software
Consortium distribution server, together with MD5 checksum
and GnuPG cryptographic signature files useful for verifying
the integrity of the tarball.
$ md5sum -v -c cds-invenio-0.99.1.tar.gz.md5
Verify MD5 checksum.
$ gpg --verify cds-invenio-0.99.1.tar.gz.sig cds-invenio-0.99.1.tar.gz
Verify GnuPG cryptographic signature. Note that you may
first have to import my public key into your keyring, if you
haven't done that already:
$ gpg --keyserver wwwkeys.eu.pgp.net --recv-keys 0xBA5A2B67
The output of the gpg --verify command should then read:
Good signature from "Tibor Simko <tibor@simko.info>"
You can safely ignore any trusted signature certification
warning that may follow after the signature has been
successfully verified.
$ tar xvfz cds-invenio-0.99.1.tar.gz
Untar the distribution tarball.
$ cd cds-invenio-0.99.1
Go to the source directory.
$ ./configure
Configure CDS Invenio software for building on this specific
platform. You can use the following optional parameters:
--prefix=/opt/cds-invenio
Optionally, specify the CDS Invenio general
installation directory (default is /opt/cds-invenio).
It will contain command-line binaries and program
libraries containing the core CDS Invenio
functionality, but also store web pages, runtime log
and cache information, document data files, etc.
Several subdirs like `bin', `etc', `lib', or `var'
will be created inside the prefix directory to this
effect. Note that the prefix directory should be
chosen outside of the Apache htdocs tree, since only
one its subdirectory (prefix/var/www) is to be
accessible directly via the Web (see below).
Note that CDS Invenio won't install to any other
directory but to the prefix mentioned in this
configuration line.
--with-python=/opt/python/bin/python2.4
Optionally, specify a path to some specific Python
binary. This is useful if you have more than one
Python installation on your system. If you don't set
this option, then the first Python that will be found
in your PATH will be chosen for running CDS Invenio.
--with-mysql=/opt/mysql/bin/mysql
Optionally, specify a path to some specific MySQL
client binary. This is useful if you have more than
one MySQL installation on your system. If you don't
set this option, then the first MySQL client
executable that will be found in your PATH will be
chosen for running CDS Invenio.
--with-clisp=/opt/clisp/bin/clisp
Optionally, specify a path to CLISP executable. This
is useful if you have more than one CLISP
installation on your system. If you don't set this
option, then the first executable that will be found
in your PATH will be chosen for running CDS Invenio.
--with-cmucl=/opt/cmucl/bin/lisp
Optionally, specify a path to CMUCL executable. This
is useful if you have more than one CMUCL
installation on your system. If you don't set this
option, then the first executable that will be found
in your PATH will be chosen for running CDS Invenio.
--with-sbcl=/opt/sbcl/bin/sbcl
Optionally, specify a path to SBCL executable. This
is useful if you have more than one SBCL
installation on your system. If you don't set this
option, then the first executable that will be found
in your PATH will be chosen for running CDS Invenio.
--with-openoffice-python
Optionally, specify the path to the Python interpreter
embedded with OpenOffice.org. This is normally not
contained in the normal path. If you don't specify this
it won't be possible to use OpenOffice.org to convert from and
to Microsoft Office and OpenOffice.org documents.
This configuration step is mandatory. Usually, you do this
step only once.
(Note that if you are building CDS Invenio not from a
released tarball, but from the Git sources, then you have to
generate the configure file via autotools:
$ sudo aptitude install automake1.9 autoconf
$ aclocal-1.9
$ automake-1.9 -a
$ autoconf
after which you proceed with the usual configure command.)
$ make
Launch the CDS Invenio build. Since many messages are printed
during the build process, you may want to run it in a
fast-scrolling terminal such as rxvt or in a detached screen
session.
During this step all the pages and scripts will be
pre-created and customized based on the config you have
edited in the previous step.
Note that on systems such as FreeBSD or Mac OS X you have to
use GNU make ("gmake") instead of "make".
$ make install
Install the web pages, scripts, utilities and everything
needed for CDS Invenio runtime into respective installation
directories, as specified earlier by the configure command.
Note that if you are installing CDS Invenio for the first
time, you will be asked to create symbolic link(s) from
Python's site-packages system-wide directory(ies) to the
installation location. This is in order to instruct Python
where to find CDS Invenio's Python files. You will be
hinted as to the exact command to use based on the
parameters you have used in the configure command.
$ make install-jsmath-plugin ## optional
This will automatically download and install in the proper
place jsMath, a Javascript library to render LaTeX formulas
in the client browser.
Note that in order to enable the rendering you will have to
set the variable CFG_WEBSEARCH_USE_JSMATH_FOR_FORMATS in
invenio-local.conf to a suitable list of output format
codes. For example:
CFG_WEBSEARCH_USE_JSMATH_FOR_FORMATS = hd,hb
$ make install-jquery-plugins ## optional
This will automatically download and install in the proper
place jQuery and related plugins. They are used for AJAX
applications such as the record editor.
Note that `unzip' is needed when installing jquery plugins.
$ make install-fckeditor-plugin ## optional
This will automatically download and install in the proper
place FCKeditor, a WYSIWYG Javascript-based editor (e.g. for
the WebComment module).
Note that in order to enable the editor you have to set the
CFG_WEBCOMMENT_USE_FCKEDITOR to True.
$ make install-pdfa-helper-files ## optional
This will automatically download and install in the proper
place the helper files needed to create PDF/A files out of
existing PDF files.
2b. Configuration
-----------------
Once the basic software installation is done, we proceed to
configuring your Invenio system.
$ sudo chown -R www-data.www-data /opt/cds-invenio
For the sake of simplicity, let us assume that your CDS
Invenio installation will run under the `www-data' user
process identity. The above command changes ownership of
installed files to www-data, so that we shall run everything
under this user identity from now on.
For production purposes, you would typically enable Apache
server to read all files from the installation place but to
write only to the `var' subdirectory of your installation
place. You could achieve this by configuring Unix directory
group permissions, for example.
$ sudo -u www-data emacs /opt/cds-invenio/etc/invenio-local.conf
Customize your CDS Invenio installation. Please read the
'invenio.conf' file located in the same directory that
contains the vanilla default configuration parameters of
your CDS Invenio installation. If you want to customize
some of these parameters, you should create a file named
'invenio-local.conf' in the same directory where
'invenio.conf' lives and write there only the customizations
that you want to be different from the vanilla defaults.
Here is a minimalist example what you would put there:
$ cat /opt/cds-invenio/etc/invenio-local.conf
[Invenio]
CFG_SITE_URL = http://your.site.com
CFG_SITE_SECURE_URL = https://your.site.com
CFG_SITE_ADMIN_EMAIL = john.doe@your.site.com
CFG_SITE_SUPPORT_EMAIL = john.doe@your.site.com
You should override at least the parameters from the top of
invenio.conf file in order to define some very essential
runtime parameters such as the visible URL of your document
server (look for CFG_SITE_URL and CFG_SITE_SECURE_URL), the
database credentials (look for CFG_DATABASE_*), the name of
your document server (look for CFG_SITE_NAME and
CFG_SITE_NAME_INTL_*), or the email address of the local CDS
Invenio administrator (look for CFG_SITE_SUPPORT_EMAIL and
CFG_SITE_ADMIN_EMAIL).
The CDS Invenio system will then read both the default
invenio.conf file and your customized invenio-local.conf
file and it will override any default options with the ones
you have specifield in your local file. This cascading of
configuration parameters will ease your future upgrades.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --update-all
Make the rest of the Invenio system aware of your
invenio-local.conf changes. This step is mandatory each
time you edit your conf files.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-tables
If you are installing CDS Invenio for the first time, you
have to create database tables.
Note that this step checks for potential problems such as
the database connection rights and may ask you to perform
some more administrative steps in case it detects a problem.
Notably, it may ask you to set up database access
permissions, based on your configure values.
If you are installing CDS Invenio for the first time, you
have to create a dedicated database on your MySQL server
that the CDS Invenio can use for its purposes. Please
contact your MySQL administrator and ask him to execute the
commands this step proposes you.
At this point you should now have successfully completed the
"make install" process. We continue by setting up the
Apache web server.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-webstat-conf
Load the configuration file of webstat module. It will create
the tables in the database for register customevents, such as
basket hits.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-apache-conf
Running this command will generate Apache virtual host
configurations matching your installation. You will be
instructed to check created files (usually they are located
under /opt/cds-invenio/etc/apache/) and edit your httpd.conf
to activate Invenio virtual hosts.
If you are using Debian GNU/Linux ``Lenny'' or later, then
you can do the following to create your SSL certificate and
to activate your Invenio vhosts:
## make SSL certificate:
$ sudo aptitude install ssl-cert
$ sudo mkdir /etc/apache2/ssl
$ sudo /usr/sbin/make-ssl-cert /usr/share/ssl-cert/ssleay.cnf \
/etc/apache2/ssl/apache.pem
## add Invenio web sites:
$ sudo ln -s /opt/cds-invenio/etc/apache/invenio-apache-vhost.conf \
/etc/apache2/sites-available/invenio
$ sudo ln -s /opt/cds-invenio/etc/apache/invenio-apache-vhost-ssl.conf \
/etc/apache2/sites-available/invenio-ssl
## disable Debian's default web site:
$ sudo /usr/sbin/a2dissite default
## enable Invenio web sites:
$ sudo /usr/sbin/a2ensite invenio
$ sudo /usr/sbin/a2ensite invenio-ssl
## enable SSL module:
$ sudo /usr/sbin/a2enmod ssl
## if you are using xsendfile module, enable it too:
$ sudo /usr/sbin/a2enmod xsendfile
If you are using another operating system, you should do the
equivalent, for example edit your system-wide httpd.conf and
put the following include statements:
Include /opt/cds-invenio/etc/apache/invenio-apache-vhost.conf
Include /opt/cds-invenio/etc/apache/invenio-apache-vhost-ssl.conf
Note that you may need to adapt generated vhost file
snippets to match your concrete operating system specifics.
Note also that you may want to tweak the generated example
configurations, especially with respect to the
WSGIDaemonProcess parameters. E.g. increase the `processes'
parameter if you have lots of RAM and many concurrent users
accessing your site in parallel.
$ sudo /etc/init.d/apache2 restart
Please ask your webserver administrator to restart the
Apache server after the above "httpd.conf" changes.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --create-demo-site
This step is recommended to test your local CDS Invenio
installation. It should give you our "Atlantis Institute of
Science" demo installation, exactly as you see it at
<http://invenio-demo.cern.ch/>.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --load-demo-records
Optionally, load some demo records to be able to test
indexing and searching of your local CDS Invenio demo
installation.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-unit-tests
Optionally, you can run the unit test suite to verify the
unit behaviour of your local CDS Invenio installation. Note
that this command should be run only after you have
installed the whole system via `make install'.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-regression-tests
Optionally, you can run the full regression test suite to
verify the functional behaviour of your local CDS Invenio
installation. Note that this command requires to have
created the demo site and loaded the demo records. Note
also that running the regression test suite may alter the
database content with junk data, so that rebuilding the
demo site is strongly recommended afterwards.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --run-web-tests
Optionally, you can run additional automated web tests
running in a real browser. This requires to have Firefox
with the Selenium IDE extension installed.
<http://en.www.mozilla.com/en/firefox/>
<http://selenium-ide.openqa.org/>
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --remove-demo-records
Optionally, remove the demo records loaded in the previous
step, but keeping otherwise the demo collection, submission,
format, and other configurations that you may reuse and
modify for your own production purposes.
$ sudo -u www-data /opt/cds-invenio/bin/inveniocfg --drop-demo-site
Optionally, drop also all the demo configuration so that
you'll end up with a completely blank CDS Invenio system.
However, you may want to find it more practical not to drop
the demo site configuration but to start customizing from
there.
$ firefox http://your.site.com/help/admin/howto-run
In order to start using your CDS Invenio installation, you
can start indexing, formatting and other daemons as
indicated in the "HOWTO Run" guide on the above URL. You
can also use the Admin Area web interfaces to perform
further runtime configurations such as the definition of
data collections, document types, document formats, word
indexes, etc.
$ sudo ln -s /opt/cds-invenio/etc/bash_completion.d/inveniocfg \
/etc/bash_completion.d/inveniocfg
Optionally, if you are using Bash shell completion, then
you may want to create the above symlink in order to
configure completion for the inveniocfg command.
Good luck, and thanks for choosing CDS Invenio.
- CDS Development Group
<cds.support@cern.ch>
<http://cdsware.cern.ch/>
c4science · Help