Homec4science
Diffusion invenio-infoscience (maint-2.0) Browse

R3600/invenio/legacy/bibsort/doc/admin/bibsort-admin-guide.webdoc
aa232da412f9maint-2.0

Last Change
Raw File

bibsort-admin-guide.webdoc
View Options

This document is not UTF8. It was detected as EUC-JP and converted to UTF8 for display.
## -*- mode: html; coding: utf-7; -*-
## This file is part of Invenio.
## Copyright (C) 2010, 2011, 2012 CERN.
##
## Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
<!-- WebDoc-Page-Title: BibSort Admin Guide -->
<!-- WebDoc-Page-Navtrail: <a class="navtrail" href="<CFG_SITE_URL>/help/admin<lang:link/>">_(Admin Area)_</a> -->
<!-- WebDoc-Page-Revision: $Id$ -->
<h2>Contents</h2>
<strong>1. <a href="#1">Overview</a></strong><br />
<strong>2. <a href="#2">Configuring BibSort</a></strong><br />
<strong>3. <a href="#3">Running BibSort</a></strong><br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.1. <a href="#3.1">Rebalancing</a><br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.2. <a href="#3.2">Inserting/Updating/Deleting records in BibSort</a><br />
<strong>4. <a href="#4">Impact on the sorted search results</a></strong><br />
<a name="1"></a><h2>1. Overview</h2>
<p>BibSort main goal is to make the sorting of search results faster.
It does this by creating several sorting buckets (that hold recids) that are then loaded by
the search_engine and cached.</p>
<p> BibSort module is active if the search_engine is using the sorting buckets to fast sort
the search results. BibSort module can be deactivated by setting the <code>CFG_BIBSORT_BUCKETS=0</code>
in the <code>invenio.conf</code> file. Also, if <code>bsrMETHOD</code> table does not contain any
data, it also means that the BibSort module is not active. The search engine will look into the BibSort
data structures to see if the method that was requested to sort the search results exists or not. If
it does not exist, then the old style sorting function (using bibxxx tables) will be used.</p>
<a name="2"></a><h2>2. Configuring BibSort</h2>
<p>Currently there is no web interface for configuring this module. All
the configuration is done via a configuration file. The location of this file is:
<pre>CFG_ETCDIR/bibsort/bibsort.cfg</pre></p>
<p>Each sorting method has a section in this config file, that looks like this:
<pre>
[sort_field_1]
name = title
washer = sort_alphanumerically_remove_leading_articles
definition = FIELD: title
</pre>
Each section of the file corresponds to a method.
<ul>
<li>The <u>name</u> property holds the name of the sorting method, that should be unique.</li>
<li>The <u>washer</u> property holds the name of the function that will process
the values of the records, for this method, before they get sorted (in this case,
it will remove the leading articles). This functions are implemented in
<i>bibsort_washer.py</i>. <br/>One can also set the washer to 'NOOP'
which will mean that no washer should be applied to the data before the sorting.<br/>
Bibsort supports <b>string collation</b>. There is the possibility of selecting a specific
language for this, by adding at the end of the washer ':ln', where 'ln' is the language
according to which the string comparison should be done.<br/>
Ex: "washer = sort_alphanumerically_remove_leading_articles:fr" means that the string comparison
would be done using the French rules.
<li>The <u>definition</u> property holds the name of the place from where the data should be
taken for this sorting method. There are several options for defining this property:</li>
<ul>
<li><code>definition = RNK: method_name</code> means that the data should be taken from the rnkMETHODDATA table, based
on the method_name (the method_name should correspond to a method in rnkMETHOD table)</li>
<li><code>definition = MARC: marc1,marc2,marc3..</code> means that BibSort will sequentially look in all
the MARC fields (bibxxx tables) and retrieve the data (the order of the MARC fields is important, since
for a given record, BibSort will keep the value from the first MARC field that has data).</li>
<li><code>definition = FIELD: foo</code> is similar with the above option, but for cases were we already have
logical fields defined in Invenio, BibSort can look into them in order to retrieve the list of MARC fields
that need to be queried.</li>
</ul>
</ul></p>
<p>For adding a new sorting method, one needs to add a new section to the <code>bibsort.cfg</code> file.
Once this is done, the config file needs to be loaded into the database:
<pre>
$ ./bibsort --load-config
</pre>
Similar, for deleting a method, one needs to remove the corresponding section from the
<code>bibsort.cfg</code> file, and load the config into the database.
<br/>
To dump the configuration from the database into a file:
<pre>
$ ./bibsort --dump-config
</pre>
</p>
<a name="3"></a><h2>3. Running BibSort</h2>
<p>There are several command line instructions that can be used in order to update the
BibSort data. For each instruction, one can define the methods and the records that
the command should run on, like this:
<pre>
$ ./bibsort --methods=method1,method2 --recids=4,7-17,23,1
</pre>
If these options will be let empty it will mean that the bibsort operations will run on
all the defined methods, and either on all the records existing in the database,
or on the all updated records (depending on the operation, see 3.1 and 3.2).
</p>
<a name="3.1"></a><h3>3.1. Rebalancing</h3>
<p>Rebalancing is the operation that will redo from scratch the sorting and recreate the
sorting buckets. This should be performed once at the beginning and then maybe once per day,
to be sure that the database is in complete sync with the BibSort data structures, and also,
to be sure that the buckets are balanced (Imagine a big upload of new records, that will
have the same publication year. All these records will be added to the same bucket for the
'publication date' method, making it much bigger then the others, and slower to perform any data
calculations on it, including intersecting with the search engine output). If you have a clear
idea of how the data is changing during one day, you can set up the rebalancing only for several
methods, that contain data that is frequently updated.</p>
<pre>
$ ./bibsort -R [--methods=method1,method2]
</pre>
<a name="3.2"></a><h3>3.2. Inserting/Updating/Deleting records in BibSort</h3>
<p>Inserting/Updating/Deleting records in BibSort is done via the update-sorting operation.
Theoretically, this operation should run at short intervals, and for the benefit of the user
it would be good to run after BibIndex, so that the updates can be viewed as soon as possible.
If no methods are defined it will run for all the methods defined in <code>bibsort.cfg</code>.
But, if you have a good overview of the nature of the changes in the data during a period of time,
the update-sorting can run more frequently for some methods (like sort by year or sort by title)
or less frequently (like sort by most cited, since the citation dictionaries are not updated so frequently).
Defining the recids, will result in the update-sorting to run only on those records. If no records are
defined bibsort will grab all the modified records since its last run. Since for ranking methods it will
anyway grab all the data, update-sorting for a ranking method is basically a rebalancing.
</p>
<pre>
$ ./bibsort -S [--methods=method1,method2] [--recids=4,7-17,23,1]
</pre>
<a name="4"></a><h2>4. Impact on the sorted search results</h2>
<p>Using the BibSort functionality will have the following impact on the 'Sort by' functionality of Invenio:
<ul>
<li>Sorting will no longer need to be limited to <code>CFG_WEBSEARCH_NB_RECORDS_TO_SORT</code> currently set to 200.</li>
<li>Not all the search results will be sorted, but only those up to <code>jrec成</code>, so only up to those 'seen' by the user.
This means that using <code>of=id</code> to retrieve the list of recids will not give the full list of recids in case
these also need to be sorted.</li>
<li>If in the search results there are records that have no value for the selected sorting method, they will be added at the end,
in the order of their insertion date. This records might have just been inserted, this is why it would be important to keep
them in the sorted output.</li>
</ul></p>
c4science · Help