Homec4science
Diffusion invenio-infoscience (iugw2017) Browse

R3600/modules/bibsched/doc/admin/bibsched-admin-guide.webdoc
826cb552c2a0iugw2017

Last Change
Raw File

bibsched-admin-guide.webdoc
View Options

## -*- mode: html; coding: utf-8; -*-
## This file is part of Invenio.
## Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: BibSched Admin Guide -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/admin<lang:link/>">_(Admin Area)_</a> -->
<!-- WebDoc-Page-Revision: $Id$ -->
<h2>Overview</h2>
<p>BibSched -- the bibliographic task scheduler -- is central unit of the
system that allows all other modules to access the bibliographic
database in a controlled manner, preventing sharing violation threats
and assuring the coherent execution of the database update tasks. The
module comes with an administrative interface that allows to monitor
the task queue including various possibilities of a manual
intervention, for example to re-schedule queued tasks, change the task
order, etc.</p>
<p>You can run the administrative interface by doing:</p>
<blockquote>
<pre>
$ bibsched
</pre>
</blockquote>
<p>Note that in general you should run bibsched with the same rights of the
Apache user of your system.</p>
<p>The <code>bibsched</code> can run in two modes: <em>automatic</em> and <em>manual</em>. In
the automatic mode, it will execute tasks automatically as they arrive in
the waiting queue. In the manual mode, the administrator has to
launch the tasks manually.</p>
<h2>Bibsched graphical interface</h2>
<p>bibsched interface is text mode graphical interface to display running tasks.
It has three views, one for listing done tasks, one for scheduled/running/failed
tasks and a third one for displaying archived tasks. You can switch among these
three views by pressing respectively "<tt>1</tt>", "<tt>2</tt>" or "<tt>3</tt>".</p>
<p>With the harrows you can move from one task to the other</p>
<p>By pressing "<tt>O</tt>" you can see all the details of the selected task</p>
<p>If the task is running or is already run, you can press "l" (lower case "<tt>l</tt>")
to access the standard output produced by the task, if any. You can press "<tt>L</tt>"
(upper case "<tt>L</tt>") to access the standard error produced by the task, if any.</p>
<p>By pressing "<tt>P</tt>" you can clean the list of DONE tasks and archive/delete
them.</p>
<p>By pressing "<tt>Q</tt>" you can Quit the interface.</p>
<p>By pressing "<tt>A</tt>" you can switch from Auto to Manual mode and viceversa.</p>
<h3>Manual mode</h3>
<p>In manual mode, depending on the status of the task you are currently
selecting you're given different actions.</p>
<p>You can press "<tt>R</tt>" for running Waiting tasks</p>
<p>You can press "<tt>D</tt>" for deleting non running tasks</p>
<p>You can press "<tt>N</tt>" for changing the priority of a waiting task. (the equivalent of the UNIX renice command)</p>
<p>On a running task you can press "<tt>K</tt>" to kill the task immediately in case of emergency. "<tt>T</tt>" for stopping it cleanly. "<tt>S</tt>" for putting it temporarily to sleep. A sleeping task can be waken up by pressing "<tt>W</tt>". Note that for stopping or putting to sleep a task, a signal is sent to the given bibtask and this, in turn, will acknowledge it and decide to stop or go to sleep whenever it thinks it's safe.</p>
<p>On a failed task you can press "<tt>K</tt>" the acknowledge the error. This is necessary in case you wish to put bibsched back to automatic mode.</p>
<h3>Automatic mode</h3>
<p>In automatic mode bibsched will take care of launching tasks based on their priority and runtime schedule. The available option are only those that allow you to query a given task (see the logs and the options).</p>
<p>If you have configured bibsched to allow for the execution of concurrent bibtasks, bibsched will take care of launching compatible tasks concurrently (note that this feature is currently experimental). Bibupload tasks will always be executed in the chronological order (to preserve input consistency).</p>
<h2>Bibsched maintenance</h2>
<p>bibsched produce two log files. bibsched.log and bibsched.err, located under the usual log directory of your Invenio installation. The former will contain all the actions (either automatic of manual) that bibsched has performed. The latter will contain all the exceptional errors.</p>
<p>In case of a bibtask failing while bibsched is in automatic mode, bibsched will stop by switching to manual mode, and will send an email to the administrator (and an emergency SMS in case it is configured to do so). Note that in case of failed bibtasks, bibsched will refuse to be put back to automatic mode, until either the task is reinitialized, or deleted or the error is acknowledged.</p>
<h2>Priorities</h2>
<p>A task can be scheduled with a given <em>priority</em>, represented by an integer number. When at a given time two or more tasks might be executed, the task with higher priority will be executed first.</p>
<p>When a task is running and is not a bibupload, the scheduler will allow to run higher priority tasks that don't conflict with the former task, by first putting to sleep the former task, if the resources are not enough.</p>
<p>If a task has priority higher than 100 and there are currently other task running, conflicting with the execution of this task (because the other tasks should not run concurrently with this task), then the other tasks are stopped (unless they are bibuploads).</p>
<p>If the priority is less than -10 than the task will never be executed automatically.</p>
<p>Bibupload tasks are not affected by priority with respect to each other and will always be executed in the proper order.</p>
<h2>Task logging</h2>
<p>When executed each tasks will produced (if necessary) a couple of log files. One called <tt>bibsched_task_{task_id}.log</tt> and the other <tt>bibsched_task_{task_id}.err</tt>. In case of reschedulable task, each time the task is rescheduled it is being assigned the same task_id. That means that log information of successive execution of the given task will be appended at the end of already existing log files.</p>
<p>A log-rotation algorithm is applied when writing into the log file. By default
each log will be no bigger than 1MB. After this limit is reached the log is rotated. Note that when viewing the log file inside the bibsched monitor
interface, only the latest log will be displayed.</p>
<h2>Task concurrency</h2>
<p>A recent experimental feature of bibsched is the concurrent execution of compatible tasks. The current definition of when two tasks are considered compatible is: "If a two tasks have the same name (e.g. bibupload) then they're incompatible."</p> <p>Sometimes you might want to consider compatible two tasks even when they have the same name. For this you can add a name specification via the bibtask command line option <tt>--name</tt>. E.g. you might want to distinguish a generic bibupload from a bibupload carrying only preformatting information. For this just launch bibupload -N "bibformat", and it will be considered compatible with all the other bibuploads.</p>
<h2>Configuration</h2>
<p>Bibsched can be tweaked by adjusting some variables in the usual <tt>invenio(-local).conf</tt> file. Please refer to the documentation associated with each variable inside this file.</p>
<h2>Bibsched command line interface</h2>
<pre>
Usage: /soft/cdsweb/bin/bibsched [options] [start|stop|restart|monitor|status]
The following commands are available for bibsched:
start start bibsched in background
stop stop running bibtasks and the bibsched daemon safely
halt halt running bibsched while keeping bibtasks running
restart restart a running bibsched
monitor enter the interactive monitor
status get report about current status of the queue
purge purge the scheduler queue from old tasks
Command options:
-d, --daemon Launch BibSched in the daemon mode (deprecated, use 'start')
General options:
-h, --help Print this help.
-V, --version Print version information.
Status options:
-s, --status=LIST Which BibTask status should be considered (default is Running,waiting)
-S, --since=TIME Since how long time to consider tasks e.g.: 30m, 2h, 1d (default
is all)
-t, --tasks=LIST Comma separated list of BibTask to consider (default
is all)
Purge options:
-s, --status=LIST Which BibTask status should be considered (default is DONE)
-S, --since=TIME Since how long time to consider tasks e.g.: 30m, 2h, 1d (default
is 30 days)
-t, --tasks=LIST Comma separated list of BibTask to consider (default
is bibindex,bibreformat,webcoll,bibrank,inveniogc,bibupload,oairepositoryupdater)
</pre>
<h2>Bibtasks command line interface</h2>
<p>Each bibtask has a common command interface in addition to the proper bibtask related options.
<pre>
Scheduling options:
-u, --user=USER User name under which to submit this task.
-t, --runtime=TIME Time to execute the task. [default=now]
Examples: +15s, 5m, 3h, 2002-10-27 13:57:26.
-s, --sleeptime=SLEEP Sleeping frequency after which to repeat the task.
Examples: 30m, 2h, 1d. [default=no]
--fixed-time Avoid drifting of execution time when using --sleeptime
-I, --sequence-id=SEQUENCE-ID Sequence Id of the current process
-L --limit=LIMIT Time limit when it is allowed to execute the task.
Examples: 22:00-03:00, Sunday 01:00-05:00.
Syntax: [Wee[kday]] [hh[:mm][-hh[:mm]]].
-P, --priority=PRI Task priority (0=default, 1=higher, etc).
-N, --name=NAME Task specific name (advanced option).
General options:
-h, --help Print this help.
-V, --version Print version information.
-v, --verbose=LEVEL Verbose level (0=min, 1=default, 9=max).
--profile=STATS Print profile information. STATS is a comma-separated
list of desired output stats (calls, cumulative,
file, line, module, name, nfl, pcalls, stdname, time).
--stop-on-error In case of unrecoverable error stop the bibsched queue.
--continue-on-error In case of unrecoverable error don't stop the bibsched queue.
--post-process=BIB_TASKLET_NAME[parameters] Postprocesses the specified
bibtasklet with the given parameters between square
brackets.
Example:--post-process "bst_send_email[fromaddr=
'foo@xxx.com', toaddr='bar@xxx.com', subject='hello',
content='help']"
</pre>
</p>
<h2>BibSched Tasklets</h2>
<p>If you have very particular needs to write your self a bibtask that can
be scheduled through the bibliographic scheduler, and you are able to write
a Python function you can write a <strong>BibTaskLet</strong></p>
<p>Suppose that you have Python function:
<pre>
def foo(arg1, arg2='default'):
pass
</pre>
that you want to execute through the bibliographic scheduler. Just put such
a function in the <tt>/opt/cds-invenio/lib/python/invenio/bibsched_tasklets</tt> in a file called e.g. <tt>bst_foo.py</tt> (the <em>bst_</em> prefix and the <em>.py</em> extension are compulsory) and rename the function to <tt>bst_foo</tt> (the name of the function must be identical to the name of the file).</p>
<p>A BibTaskLet can be executed through the <tt>bibtasklet</tt> BibTask. E.g.:
<pre>
$ # To list the available bibtasklets
$ sudo -u apache /opt/cds-invenio/bin/bibtasklet -l
Available tasklets:
╔══════════════════════════════════════════════════════════════════╗
║ def bst_fibonacci(n=30) ║
╠══════════════════════════════════════════════════════════════════╣
║ ║
║ Small tasklets that prints the the Fibonacci sequence for n. ║
║ @param n: how many Fibonacci numbers to print. ║
║ @type n: int ║
║ ║
╚══════════════════════════════════════════════════════════════════╝
╔════════════════════════════════════╗
║ def bst_foo(arg1, arg2='default') ║
╠════════════════════════════════════╣
║ ║
║ No documentation. ║
║ ║
╚════════════════════════════════════╝
Broken tasklets:
$ # To schedule a bibtasklet
$ sudo -u apache /opt/cds-invenio/bin/bibtasklet -T bst_foo -a "arg1=bar"
</pre>
All the above bibtask options are available for any bibtasklet.
</p>
<p>See <tt>/opt/cds-invenio/lib/python/invenio/bibsched_tasklets/bst_fibonacci.py</tt> for an example on how a bibtasklet look like.
</p>
c4science · Help