Homec4science
Diffusion invenio-infoscience (maint-1.2) Browse

R3600/modules/webhelp/web/admin/howto/howto-run.webdoc
912b49c96c46maint-1.2

Last Change
Raw File

howto-run.webdoc
View Options

# This file is part of Invenio.
# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015 CERN.
#
# Invenio is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of the
# License, or (at your option) any later version.
#
# Invenio is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Invenio; if not, write to the Free Software Foundation, Inc.,
# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: HOWTO Run Your Invenio Installation -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/admin<lang:link/>">Admin Area</a> &gt; <a class="navtrail" href="howto">Admin HOWTOs</a> -->
<!-- WebDoc-Page-Revision: $Id$ -->
<h2>Overview</h2>
<p>This HOWTO guide intends to give you ideas on how to run your CDS
Invenio installation and how to take care of its normal day-to-day
operation.
</p>
<h2>Setting up periodical daemon tasks</h2>
<p>Many tasks that manipulate the bibliographic record database can be
set to run in a periodical mode. For example, we want to have the
indexing engine to scan periodically for newly arrived documents to
index them as soon as they enter into the system. It is the role of
the BibSched system to take care of the task scheduling and the task
execution.
</p>
<p>Periodical tasks (such as regular metadata indexing) as well as
one-time tasks (such as a batch upload of newly acquired metadata
file) are not executed straight away but are stored in the BibSched
task queue. BibSched daemon looks periodically in the queue and
launches the tasks according to their order or the date of programmed
runtime. You can consider BibSched to be a kind of cron daemon for
bibliographic tasks.
</p>
<p>This means that after having installed Invenio you will want to
have the BibSched daemon running permanently. To launch BibSched
daemon, do:
<blockquote>
<pre>
$ bibsched start
</pre>
</blockquote>
</p>
<p>To setup indexing, ranking, sorting, formatting, and collection cache updating
daemons to run periodically with a sleeping period of, say, 5 minutes:
<blockquote>
<pre>
$ bibindex -f50000 -s5m
$ bibreformat -oHB -s5m
$ webcoll -v0 -s5m
$ bibrank -f50000 -s5m
$ bibsort -s5m
</pre>
</blockquote>
</p>
<p>Note that if you are using virtual index facility, such as for
the <em>global</em> index, then you should schedule them apart:
<blockquote>
<pre>
$ bibindex -w global -f50000 -s5m
</pre>
</blockquote>
</p>
<p>It is imperative to have the above tasks run permanently in your
BibSched queue so that newly submitted documents will be processed
automatically.
</p>
<p>You may also want to set up some periodical housekeeping tasks:
<blockquote>
<pre>
$ bibrank -f50000 -R -wwrd -s14d -LSunday
$ bibsort -R -s7d -L 'Sunday 01:00-05:00'
$ inveniogc -a -s7d -L 'Sunday 01:00-05:00'
$ dbdump -s20h -L '22:00-06:00' -o/opt2/mysql-backups -n10
</pre>
</blockquote>
</p>
<p>Please consult the sections below for more details about these
housekeeping tasks.</p>
<p>There is also the possibility to setup the batch uploader daemon
to run periodically, looking for new documents or metadata files
to upload:
<blockquote>
<pre>
$ batchuploader --documents -s20m
$ batchuploader --metadata -s20m
</pre>
</blockquote>
</p>
<p>Additionally you might want to automatically generate <tt>sitemap.xml</tt>
files for your installation. For this just schedule:
<blockquote>
<pre>
$ bibexport -w sitemap -s1d
</pre>
</blockquote>
You will then need to add a line such as:
<blockquote>
<pre>
Sitemap: <CFG_SITE_URL>/sitemap-index.xml.gz
</pre>
</blockquote>
to your <tt>robots.txt</tt> file.</p>
<p>If you are using the WebLinkback module, you may want to run some of the following tasklets:
<blockquote>
<pre>
# Delete rejected, broken and pending linkbacks whose URLs is on blacklist
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaterdeleteurlsonblacklist \
-T bst_weblinkback_updater \
-a "mode=1" \
-u admin -s1d -L '22:00-05:00'
# Update page titles of new linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaternewpages \
-T bst_weblinkback_updater \
-a "mode=2" \
-u admin -s1d -L '22:00-05:00'
# Update page titles of old linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdateroldpages \
-T bst_weblinkback_updater \
-a "mode=3" \
-u admin -s7d -L '22:00-05:00'
# Update manually set page titles
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdatermanuallysettitles \
-T bst_weblinkback_updater \
-a "mode=4" \
-u admin -s7d -L '22:00-05:00'
# Detect and disable broken linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbackupdaterdetectbrokenlinkbacks \
-T bst_weblinkback_updater \
-a "mode=5" \
-u admin -s7d -L 'Sunday 01:00-05:00'
# Send notification email for all pending linkbacks
sudo -u www-data /opt/invenio/bin/bibtasklet \
-N weblinkbacknotifications \
-T bst_weblinkback_updater \
-a "mode=6" \
-u admin -s1d
</pre>
</blockquote>
</p>
<h2>Monitoring periodical daemon tasks</h2>
<p>Note that the BibSched automated daemon stops as soon as some of
its tasks end with an error. You will be informed by email about this
incident. Nevertheless, it is a good idea to inspect the BibSched
queue from time to time anyway, say several times per day, to see what
is going on. This can be done by running the BibSched command-line
monitor:
<blockquote>
<pre>
$ bibsched
</pre>
</blockquote>
</p>
<p>The monitor will permit you to stop/start the automated mode, to
delete the tasks that were wrongly submitted, to run some of the tasks
manually, etc. Note also that the BibSched daemon writes its log and
error files about its own operation, as well as on the operation of
its tasks, to the <code>/opt/invenio/var/log</code> directory.</p>
<h2>Running alert engine</h2>
<p>Invenio users may set up an automatic notification email alerts
so that they are automatically alerted about documents of their
interest by email, either daily, weekly, or monthly. It is the job of
the alert engine to do this. The alert engine has to be run every
day:
<blockquote>
<pre>
$ alertengine
</pre>
</blockquote>
</p>
<p>You may want to set up an external cron job to call
<code>alertengine</code> each day, for example:
<blockquote>
<pre>
$ crontab -l
&#35; invenio: periodically restart Apache:
59 23 * * * /usr/sbin/apachectl restart
&#35; invenio: run alert engine:
30 14 * * * /usr/bin/sudo -u apache /opt/invenio/bin/alertengine
</pre>
</blockquote>
</p>
<h2>Housekeeping task details</h2>
<h4>Housekeeping: recalculating ranking weights</h4>
<p>When you are adding new records to the system, the word frequency
ranking weights for old records aren't recalculated by default in
order to speed up the insertion of new records. This may influence a
bit the precision of word similarity searches. It is therefore
advised to expressely run bibrank in the recalculating mode once in a
while during a relatively quiet site operation day, by doing:
<blockquote>
<pre>
$ bibrank -R -w wrd -s 14d -L Sunday
</pre>
</blockquote>
You may want to do this either (i) periodically, say once per month
(see the previous section), or (ii) depending on the frequency of new
additions to the record database, say when the size grows by 2-3
percent.
</p>
<h4>Housekeeping: recalculating sorting weights</h4>
<p>It is advised to run from time to time the rebalancing of the
sorting buckets. In order to speed up the process of insertion of
new records, the sorting buckets are not being recalculated,
but new records are being added at the end of the corresponding
bucket. This might create differences in the size of each bucket
which might have a small impact on the speed of sorting.
<blockquote>
<pre>
$ bibsort -R -s 7d -L 'Sunday 01:00-05:00'
</pre>
</blockquote>
The rebalancing might be run weekly or even daily.
</p>
<h4>Housekeeping: cleaning up the garbage</h4>
<p>The tool <code>inveniogc</code> provides a garbage collector for
the database, temporary files, and the like.</p>
<p>Different temporary log and err files are created
in <code>/opt/invenio/var/log</code>
and <code>/opt/invenio/var/tmp</code> directory that is good to
clean up from time to time. The previous command could be used to
clean those files, too, via:
<blockquote>
<pre>
$ inveniogc -s 7d -d -L 'Sunday 01:00-05:00'
</pre>
</blockquote>
</p>
<p>The <code>inveniogc</code> tool can run other cleaning actions;
please refer to its help (<code>inveniogc --help</code>) for more
details.</p>
<p>Note that in the above section "Setting up periodical daemon
tasks", we have set up <code>inveniogc</code> with
argument <code>-a</code> in that example, meaning that it would run
all possible cleaning actions. Please modify this if it is not what
you want.</p>
<h4>Housekeeping: backing up the database</h4>
<p>You can launch a bibsched task called <code>dbdump</code> in order
to take regular snapshot of your database content into SQL dump files.
For example, to back up the database content
into <code>/opt2/mysql-backups</code> directory every night, keeping
at most 10 latest copies of the backup file, you would launch:</p>
<blockquote>
<pre>
$ dbdump -s 20h -L '22:00-06:00' -o /opt2/mysql-backups -n 10
</pre>
</blockquote>
<p>This will create files named
like <code>invenio-dbdump-2009-03-10_22:10:28.sql</code> in this
folder.</p>
<p>Note that you may use Invenio-independent MySQL backuping tools
like <code>mysqldump</code>, but these might generally lock all tables
during backup for consistency, hence it could happen that your site
might not be accessible during backuping time due to the user session
table being locked as well. The <code>dbdump</code> tool does not
lock all tables, therefore the site remains accessible to users while
the dump files are being created. Note that the dump files are kept
consistent with respect to the data, since <code>dbdump</code> runs
via <code>bibsched</code>, hence not allowing any other important
bibliographic task to run during the backup.</p>
<p>To load a dump file produced by <code>dbdump</code> into a running
Invenio instance later, you can use:</p>
<blockquote>
<pre>
$ bibsched stop
$ cat /opt2/mysql-backups/invenio-dbdump-2009-03-10_22\:10\:28.sql | dbexec
$ bibsched start
</pre>
</blockquote>
c4science · Help