<p>BibClassify automatically extracts keywords from fulltext documents.
The automatic assignment of keywords to textual documents has clear
benefits in the digital library environment as it aids
catalogization, classification and retrieval of documents.</p>
<a name="1.1"></a><h3>1.1 Thesaurus</h3>
<p> BibClassify performs an extraction of keywords based on the
recurrence of specific terms, taken from a controlled vocabulary. A
controlled vocabulary is a thesaurus of all the terms that are
relevant in a specific context. When a context is defined by a
discipline or branch of knowledge then the vocabulary is said to be a
<em>subject thesaurus</em>. Various existing subject thesauri can be found <a href="http://www.fbi.fh-koeln.de/institut/labor/Bir/thesauri_new/thesen.htm">here</a>.</p>
<p> A subject thesarus can be expressed in several different
formats. Different institutions/disciplines have developed different
ways of representing their vocabulary systems. BibClassify accepts thesauri in two formats:</p>
<ul>
<li> <b>Simple text</b>. <br />
This is a simple list of allowed keywords. One keyword per line. E.g. <br />
<blockquote>
<pre>
asymmetry
asymptotic behavior
ATLAS
atmosphere
</pre>
</blockquote>
</li><li> <b> RDF SKOS taxonomy </b>. <br />
This is a richer and more complex structure to describe concepts. E.g. <br />
<p> This section provides a detailed account of the phrase matching
techniques used by BibClassify to automatically extract significant terms
from fulltext documents. While reading this guide, you are advised to refer to the original
BibClassify code, mostly contained in <code>lib/python/invenio/bibclassifylib.py</code>.
This guide refers to version 2006/09/15.</p>
<p>The bulk of the extraction mechanism takes place inside the function
<code>generate_keywords_rdf</code>. This function is triggered when
BibClassify is launched with parameter <code>-K,
--taxonomy=FILENAME</code>. Let's have a look at what happens inside this function, step-by-step.
<p><span class="adminbox"> <b>NB.</b> BibClassify can also run on top of simple text thesauri
(parameter <code>-k, --thesaurus=FILENAME</code>, function
<code>generate_keywords</code>), however this mode is now deprecated
and no longer maintained.</span></p>
<a name="2"></a><h2>2. Preprocessing</h2>
<p> At the beginning of the function, various local variables are
declared. Among these,</p>
<ul>
<li><code>namespace</code>: This variable
points to the main rdf:Namespace used in the taxonomy. In the case of
RDF SKOS, this is
<code>http://www.w3.org/2004/02/skos/core#</code>. If you need to use namespaces other than this one, please modify this variable
accordingly.</li>
<li><code>delimiter</code>: This variable represents the delimiter
symbol used to separate composite keywords. For example, current HEP
taxonomy adopts ":" (e.g. <em>baryon: asymmetry</em>) whereas the SPIRES standard
adopts "," (e.g. <em>baryon, asymmetry</em>). If you intend to set up and use
composite keywords in your taxonomy, please set this variable to the desired format.
</ul>
<p>The taxonomy (<code>dictfile</code>) is stored and parsed into memory via <a href="http://rdflib.net/">RDFlib</a> by the following two lines of code:
<blockquote>
<code>
store = rdflib.Graph()<br />
store.parse(dictfile)
</code>
</blockquote></p>
<p><span class="adminbox"> <b>NB.</b> RDFLib provides very handy
libraries for RDF manipulation, however when dealing with large RDF
files, loading and parsing are by far the principal factor affecting
the performance of BibClassify. For example, when loading the HEP
taxonomy (in <code>/opt/cds-invenio/etc/bibclassify/HEP.rdf</code>:
7.4 MB, 16000 Concepts) on an Intel(R) Xeon(TM) CPU 3.06GHz the two
lines of code above take a total of <b>26 seconds</b> to complete
(over two thirds of the total execution time - 36 seconds). If
performance is your main concern, consider a faster library or working
on a pre-loaded RDF store. </span>
</p>
<p>At this point, the fulltext of the document is converted from text
to PDF (using standard linux command <code>pdftotext</code>) and stored into a
string <code>text_string</code>. This string will contain the full
document if running in slow mode or an arbitrary excerpt of the
document (about 40%) if running in fast mode. Moreover, the very
beginning of the string (10%) is stored in a variable called
<code>abstract</code>. This is done base on the assumption that
manuscripts generally contain crucial information such as title and
abstract in the very first portion of the document. This portion can
be then treated to be more relevant than the remainder of the document. Please
bear this in mind if running BibClassify on documents with different
structures or when running on heterogeneous collections.
</p>
<p> In many manuscripts, the author includes a list of pre-assigned
terms that describe the topic of the article. The last step before
keyword extraction begins is to locate these author-assigned
keywords. We try to isolate these by searching for the key phrase
<em>Keywords</em> followed by a list of terms. When found, the string
is stored into variable <code>safe_keys</code> and used later to match
BibClassify output against author assigned keywords (these are marked
in the output with an asterisk, e.g. <code>13* Hubble constant</code>)
</p>
<a name="3"></a><h2>3. Single Keyword (mkw) processing</h2>
<p> The bulk of the phrase matching operations - the extraction of the single
keywords from the fulltext - is contained in a big <code>for</code> loop. In this
loop, every RDF <code>Concept</code> is parsed, one at a time, and its
components (such as <code>prefLabel</code> and <code>altLabel</code>) matched inside the document.
</p>
<p><span class="adminbox"> <b>NB.</b> For a detailed explanation
<p><span class="adminbox"> <b>NB.</b> One could add at this point
the possibility of having combinations of more than two mkws, e.g.
<em>ckwURI : [mkw1URI, mkw2URI, mkw3URI]</em>. This is feasible, but
was not implemented at this stage, because of the performance overhead that would
be generated by the phrase matching of more complex regular expressions.</span>
</p>
<p>Once the <code>composites</code> dictionary is completed, its keys
that point to lists of two values (like the example above) are ckw
candidates. We now need to check whether they actually appear one next
to the other in the text. This is done in function
<code>makeCompPattern</code> by compiling a pair of regular
expressions: one for <em>mkw1</em> followed by <em>mkw2</em> and one
for the inverse situation, <em>mkw2</em> followed by <em>mkw1</em>. Once again,
when compiling the regex pattern we have to take extra care to treat
special cases (hyphens, short names, wildcards) accordingly. The sum
of the incidence of the two patterns in the <code>text_string</code> is
stored in a list (<code>compositesOUT</code>) that is then sorted (by
occurrence) and output.
</p>
<a name="5"></a><h2>5. Postprocessing</h2>
<p> Before presenting the results to the user, some extra filtering
occurs, primarily to refine the output keywords. The main postprocessing actions performed on the results are:</p>
<ul>
<li> Ensure that the order of the occurrence counter ([n1,n2]) for composite keywords (e.g. <em>baryon: asymmetry [7, 12]</em>) is correct.</li>
<li> Ensure that "stray" wildcard labels (e.g. hiddenLabels of composite keywords) do not cause double phrase matching.</li>
<li> Produce the desired output using the chosen ckw delimiter: the BibClassify standard (:) or the SPIRES one (,).</li>
<li> Filter out single keywords that match one into each other, e.g. if <em>magnetic</em> and <em>magnetic field</em> appear among mkws, subtract the occurrence of <em>magnetic</em> from <em>magnetic field</em>.
<p><span class="adminbox"> <b>NB.</b> One could also perform this
last post-processing step at the composite keyword level, e.g. <em>energy:
density</em> to be overridden by <em>dark energy: density</em>. This has
not been yet implemented for security reasons (the incidence of altLabels
on the ckw computation).</li></ul>
<p> The final results that are produced to the user consist of the
first <code>n</code> entries of <code>keylist</code> (the single
keywords) and the entries in <code>compositesOUT</code> (the composite
keywords). The results may be presented in text or html format,
according to the output mode chosen at the command line. Sample output (both text and html) can be found in the
thesaurus</a> to a more complex taxonomy (in <code>/opt/cds-invenio/etc/bibclassify/HEP.rdf</code>), with richer structure
and semantics, was mainly driven by the needs of the BibClassify system.
The current taxonomy oh high energy physics is expressed in the <a href="http://www.w3.org/2004/02/skos/">SKOS</a>
syntax. SKOS is a dialect of RDF and it is especially intended for the
representation of knowledge organization systems, such as thesauri,
taxonomies and basic ontologies.
<p><span class="adminbox"> <b>NB.</b> The reasons behind the adoption of SKOS, instead of other similar
knowledge organization formats - notably <a href="http://www.w3.org/TR/owl-features/">OWL</a> - are to do with the simplicity, yet completeness, of SKOS.
The SKOS language contains all the basic properties that were needed to express the taxonomy and it allows straightforward conversion from text to RDF.
If you are interested in a more detailed discussion on this matter, please check the <a
- <td><img src="<WEBURL>/img/admin/bibformat-guide-bfo_hd_rules.png" alt="1. Use Template [Picture HTML Detailed] if tag [980__a] is equal to [PICTURE] 2. Use Template [Thesis HTML detailed] if tag [980__a] is equal to [THESIS] 3. By default use [Default HTML Detailed]"/>
+ <td><img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_hd_rules.png" alt="1. Use Template [Picture HTML Detailed] if tag [980__a] is equal to [PICTURE] 2. Use Template [Thesis HTML detailed] if tag [980__a] is equal to [THESIS] 3. By default use [Default HTML Detailed]"/>
<td>
Beside is a screenshot of the "hd" or "HTML Detailed" output format.
You can see that the output format does not specify how to format the record, but
contains a set of rules which define which template must be used.<br/>
The rules are evaluated from top to bottom.
Each rule defines a condition on a field of the record, and a format template to use to
format the record if the condition matches.
Let's say that the field 980__a of the record is equal to
"Picture". Then first rules matches, and format template
<span class="formatTemplateBox">Picture HTML Detailed</span> is
used for formatting by BibFormat.<br/>
You can add, remove or edit output formats <a href="bibformatadmin.py/output_formats_manage">here</a></td>
<p>This is were you can delete, create or check output formats.
The menu at the top of the page let you go to other admininistration pages.<br/>
Click on the "Add New Output Format" button at the bottom of the page. You can then fill in some attributes
for the output format. Choose "title" as code, "Only Title" as name and "Prints only title" as description:</p>
-<img src="<WEBURL>/img/admin/bibformat-guide-bfo_attributes.png" alt="Screenshot of the Update Output Format Attributes page" />
+<img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_attributes.png" alt="Screenshot of the Update Output Format Attributes page" />
<p>Leave other fields blank, and click on the button "Update Output format Attributes".<br/> You are then
redirected to the rules editor. Notice the menu at the top which let you close the editor, change the attributes again
and check the output format. However do not click on these links before saving your modification of rules!</p>
-<img src="<WEBURL>/img/admin/bibformat-guide-bfo_rules.png" alt="Output format menu" />
+<img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_rules.png" alt="Output format menu" />
<p>As our format does not need to have a different behaviour depending on the record, we do not need to add new rules to the format. You just need to select a format template in the "By default use" list. However we first have to create our special format template that only print titles. So close the editor using the menu at the top of the page, and in the menu that just appeared instead, click on "Manage Format Templates". In a similar way to output formats, you see the list of format templates.</p>
Click on the "Add New Format Template" button at the bottom of the page. As for the output format, fill in the attributes of the template with name "Title" and any relevant description.
</p>
-<img src="<WEBURL>/img/admin/bibformat-guide-bft_attributes.png" alt="update format template attributes"/>
+<img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bft_attributes.png" alt="update format template attributes"/>
<p>Click on the "Update Output Format Attributes" button. You are redirected to the template editor. The editor is divided in three parts. The upper left part contains the code of the template. The bottom part is a preview of the template. The part on the right side is a short remainder of the format elements you can use in you template. You can hide this documentation by clicking on "Hide Documentation".</p>
<p>The above screenshot shows the template code already filled in. It calls the <code>BFE_TITLE</code> element. If you do not know the name of the element you want to call, you can search for it using the embedded documentation search. You can try to add other elements into your template, or write some HTML formatting.</p>
<p>When you are satisfied with your template, click on the save button, close the editor and go back to the "Only titles" output format rules editor. There select the template you have just created in the "Use by default" menu and save the ouput format and you are done. </p>
<p>This tutorial does not cover all aspects of the management of formats (For example "Knowledge bases" or internationalization). It also does not show all the power of output formats, as the one we have created simply call a template. However you have seen enough to configure BibFormat trough the web interface. Read the sections below to learn more about it.</p>
<h3><a name="administerWebFile">1.3 Administer Through the Web Interface or Through the Configuration files</a></h3>
<p>BibFormat can be administered in two ways. The first way is to use the provided web interface. It should be the most
convenient way of doing for most users. The web interface is simple to use and provides great tools to manage your formats. Its only limitation concerns the format elements, which cannot be modified using it (But the web interface provide a dynamically generated documentation of your elements). <br/>
The other way to administer BibFormat is to directly modify the configuration files using your preferred text editor. This way of doing can bring much power to advanced users, but requires an access to the server's files. It also requires that the user double-check his modifications, or use the web interface to ensure the validity and correctness of his formats.</p>
<p>In this manual we will show both ways of doing. For each explication we show first how to do it through the web interface, then how to do it by manipulating the configuration files. Non-power users can stop reading as soon as they encounter the text "For developers and adventurers only".</p>
<p>We generally recommend to use the web interface, excepted for writing
bibliographic records, you cannot specify manually for each of them
how it should be formatted. This is why you can define rules that will
allow BibFormat to understand which kind of formatting to apply to a given
record. You define this set of rules in what is called an "output
format".</p>
<p>You can have different output formats, each with its own characteristics.
For example you certainly want that when multiple bibliographic records are
displayed at the same time (as it happens in search results), only
short versions are shown to the user , while a detailed record is
preferable when a single record is displayed, whatever the type of the record.<br/>
You might also want to
let your users decide which kind of output they want. For example you
might need to display HTML for regular web browsing, but would also
give a BibTeX version of the bibliographic reference for direct
inclusion in a LaTeX document.</p>
<p>To summarize, an output format groups similar kind of formats, specifying which kind
of formatting has to be done, but not how it has to be done.</p>
<h3><a name="addOutputFormat">2.1 Add an Output Format</a></h3>
<p>To add a new output format, go to the <a href="bibformatadmin.py/output_formats_manage">Manage Output Formats</a> page and click on the "Add New Output Format" button at the bottom of the page. The format has been created. You can then specify the attributes of the output format. See <a href="#attrsOutputFormat">Edit the Attributes of an Output Format</a> to learn more about it.<p>
<strong>For developers and adventurers only:</strong>
<p>Alternatively you can directly add a new output format file into the
/etc/bibformat/outputs/ directory of your CDS Invenio installation, if you have
access to the server's files. Use the format extension .bfo for your file.</p>
<p>You should also check that user <code>www-data</code> has read/write access to the file,
if you want to be able to modify the rules through the web interface.</p>
<h3><a name="removeOutputFormat">2.2 Remove an Output Format</a></h3>
<p>To remove an output format, go to the <a href="bibformatadmin.py/output_formats_manage">Manage Output Formats</a> page and click on the "Delete" button facing the output format you want to delete. If you cannot click on the button (the button is not enabled), this means that you do not have sufficent priviledge to do so (Format is protected. Contact the administrator of the system).</p>
<strong>For developers and adventurers only:</strong>
<p>You can directly remove an output format from the /etc/bibformat/outputs/ directory of your CDS Invenio installation.
However you must make sure that it is removed from the tables <code>format</code> and <code>formatname</code> in the database, so that other modules know that it is not longer available.</p>
<h3><a name="rulesOutputFormat">2.3 Edit the Rules of an Output Format</a></h3>
<p>When you create a new output format, you can at first only specify the default template,
that is the one which is used when all rules fail. In the case of a basic output format,
this is enough. You can however add other rules, by clicking on the "Add New Rule" button.<br/>
Once you have added a rule, you can fill it with a condition, and a template that should be used
if the condition is true. For example the rule</p>
- <img src="<WEBURL>/img/admin/bibformat-guide-bfo_edit_rule.png" alt="Rule: Use template [Picture HTML Detailed] if field [980__a] is equal to [PICTURE]"/>
+ <img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_edit_rule.png" alt="Rule: Use template [Picture HTML Detailed] if field [980__a] is equal to [PICTURE]"/>
<p> will use template named "Picture HTML Detailed" if the field <code>980__a</code> of the record to format is equal to "Picture".
Note that text "PICTURE" will match any letter case like "picture" or "Picture".
Leading and trailing spaces are ignored too (" Picture " will match "PICTURE").
<br/><b>Tips:</b> you can use a regular expression as text. For example "PICT.*" will match "pictures"
and "PICTURE".</p>
- <p><div style="float:right"><img src="<WEBURL>/img/admin/bibformat-guide-bfo_edit_rule2.png" alt="Reorder rules using arrows"/></div>
+ <p><div style="float:right"><img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_edit_rule2.png" alt="Reorder rules using arrows"/></div>
The above configuration will use format template "Default HTML Detailed" if all above rules fail (in that case
if field 980__a is different from "PICTURE"). If you have more rules, you decide in which order the conditions are evaluated. You can reorder rules by clicking on the small arrows on the left of the rules.
</p>
<p>Note that when you are migrating your output formats from the old PHP BibFormat, you might not have translated all the formats to which your output formats refers. In that case you should use <code>defined in old BibFormat</code> option in the format templates menu, to make BibFormat understand that a match for this rule must trigger a call to the <i>Behaviour</i> of the old BibFormat. See section on <a href="#runSideBySide">Run old and new formats side by side</a> for more details on this.</p>
<strong>For developers and adventurers only:</strong>
<p>To write an output format, use the following syntax:<br/>
First you
define which field code you put as the conditon for the rule.
You suffix it with a column. Then on next lines, define the values of
the condition, followed by --- and then the filename of the template
to use:</p>
<pre>
tag 980__a:
PICTURE --- PICTURE_HTML_BRIEF.bft
PREPRINT --- PREPRINT_HTML_BRIEF.bft
PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
</pre>
<p>
This means that if value of field 980__a is equal to PICTURE, then we
will use format template PICTURE_HTML_BRIEF.bft. Note that you must
use the filename of the template, not the name. Also note that spaces
at the end or beginning are not considered. On the following lines,
you can either put other conditions on tag 980__a, or add another tag on
which you want to put conditions.</p>
<p>At the end you can add a default condition:</p>
<pre>
default: PREPRINT_HTML_BRIEF.bft
</pre>
<p>which means that if no condition is matched, a format suitable for
Preprints will be used to format the current record.<p>
<p>The output format file could then look like this:</p>
<pre>
tag 980__a:
PICTURE --- PICTURE_HTML_BRIEF.bft
PREPRINT --- PREPRINT_HTML_BRIEF.bft
PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
tag 8560_f:
.*@cern.ch --- SPECIAL_MEMBER_FORMATTING.bft
default: PREPRINT_HTML_BRIEF.bft
</pre>
<p> You can add as many rules as you want. Keep in mind that they are read
in the order they are defined, and that only first rule that
matches will be used.
Notice the condition on tag 8560_f: it uses a regular expression to
match any email address that ends with @cern.ch (the regular
expression must be understandable by Python)
</p>
<h3><a name="attrsOutputFormat">2.4 Edit the Attributes of an Output Format</a></h3>
<p>An output format has the following attributes:
<ul>
<li><code>code</code>: a short identifier that is used to identify the output format. It must be unique and contain a maximum of 6 letters. Note that the <b>code is not case sensitive</b> ("HB" is equal to "hb").</li>
<li><code>content type</code>: this is the content type of the format, specified in Mime. For example if you were to produce an Excel output, you could use <code>application/ms-excel</code> as content type. If a content type is specified, CDS Invenio will not print the usual header and footerfor the page, but will trigger a download in the client's browser when viewing the page (Unless the browser handles this content type).</li>
<li><code>name</code>: a generic name to display in the interface for this output format.</li>
<li>(*) <code>name</code>: internationalized names for the output format, used for displaying localized name in the search interface.</li>
<li><code>description</code>: an optional description for the output format.</li>
</ul></p>
<p><b>Please read this information regarding output format codes:</b>
There are some reserved codes that you should not use, or at least be aware of when choosing a code for your
output format. The table below summarizes these special words:
</p>
<p align="center">
<small>
<table width="40%" border="1">
<tr>
<th>Code</th><th>Purpose</th>
</tr><tr>
<td>HB</td><td>Used for displaying list of results of a search.</td>
</tr><tr>
<td>HD</td><td>Used when no format is specified when viewing a record.</td>
</tr><tr>
<td>HM</td><td>Used for Marc output. The format is special in the sense that it filters
fields to display according to the 'ot' GET parameter of the HTTP request.</td>
</tr><tr>
<td>Starting with letter 't'</td><td>Used for displaying the value of the field specified by the 'ot' GET parameter of the HTTP request.</td>
</tr><tr>
<td>Starting with 3 digits</td><td>Used for displaying the value of the field specified by the digits.</td>
</tr>
</table>
</small>
</p>
<strong>For developers and adventurers only:</strong>
<p>Excepted for the code, output format attributes cannot be changed in the output format file. These
attributes are saved in the database. As for the <code>code</code>, it is the name of the output format file,
without its <code>.bfo</code> extension. If you change this name, do not forget to propagate the modification in the database.</p>
<h3><a name="dependenciesOutputFormat">2.5 Check the Dependencies an Output Format</a></h3>
<p>To check the dependencies of an output format on format templates, format elements and tags,
go to the <a href="bibformatadmin.py/output_formats_manage">Manage Output Formats</a> page, click on
the output format you want to check, and then in the menu click on "Check Dependencies".</p>
<ul> <li>the format templates which might be called by the rules of the output format</li>
<li>the elements used in each of these templates</li>
<li>the Marc tags involved in these elements</li>
</ul>
Note that some Marc tags might be omitted.</p>
<h3><a name="validityOutputFormat">2.6 Check the Validity an Output Format</a></h3>
<p>To check the validity of an output format, simply go to the <a href="bibformatadmin.py/output_formats_manage">Manage Output Formats</a> page, and look at the column 'status' for the output format you want to check. If message "Ok" is there,
then no problem was found with the output format. If message 'Not Ok' is in the column, click on it to see
the problems that have been found for the output format.</p>
<h2><a name="formatTemplates">3. Format Templates</a></h2>
<p>A format template defines how a record should be formatted. For example it specifies which fields of the record are to be displayed, in which order and with which visual attributes. Basically the format template is written in HTML, so that it is easy for anyone to edit it. BibFormat also has support for XSLT for formatting. Read more <a href="#xslFormatTemplate">about XSL format templates here</a>.</p>
<h3><a name="addFormatTemplate">3.1 Add a Format Template</a></h3>
<p>To add a new format template, go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page and click on the "Add New Format Template" button at the bottom of the page. The format has been created. You can then specify the attributes of the format template, or ask to make a copy of an existing format.
See <a href="#attrsFormatTemplate">Edit the Attributes of a Format Template</a> to learn more about editing the attributes.<p>
<strong>For developers and adventurers only:</strong>
<p>Alternatively you can directly add a new format template file into the
/etc/bibformat/format_templates/ directory of your CDS Invenio installation, if you have
access to the server's files. Use the format extension .bft for your file.</p>
<p>You should also check that user <code>www-data</code> has read/write access to the file,
if you want to be able to modify the code and the attributes of the template through the web interface.</p>
<h3><a name="removeFormatTemplate">3.2 Remove a Format Template</a></h3>
<p>To remove a format template, go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page and click on the "Delete" button facing the format template you want to delete. If you cannot click on the button (the button is not enabled), this means that you do not have sufficent priviledge to do so (Format is protected. Contact the administrator of the system).</p>
<strong>For developers and adventurers only:</strong>
<p>You can directly remove the format template from the /etc/bibformat/format_templates/ directory of your CDS Invenio installation.</p>
<h3><a name="codeFormatTemplate">3.3 Edit the Code of a Format Template</a></h3>
<p>You can change the formatting of records by modifying the code of a template.
<p>To edit the code of a format template
go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page. Click on
the format template you want to edit to load the template editor.</p>
<p>The format template editor contains three panels. The left upper panel is the code editor. This is were
you write the code that specifies the formatting of a template. The right-most panel is a short documentation
on the "bricks" you can use in your format template code. The panel at the bottom of the page allows you to preview the template. </p>
<p>The first thing you have to know before editing the code is that everything you write in the
code editor is printed as such by BibFormat. Well almost everything (as you will discover later).</p>
<p>For example if you write "My Text", then for every record the output will be "My Text". Now let's say
you write "<b>My Text</b>": the output will still be "<b>My Text</b>", but as we display in a web browser, it will look like
"<b>My Text</b>" (The browser interprets the text inside tags <b></b> as "bold". Also note that the look may depend on the CSS style of your page).</p>
<p>Basically it means that you can write HTML to do the formatting. If you are not experienced with HTML you can use an HTML editor to create your layout, and the copy-paste the HTML code inside the template.</p>
<p>Do not forget to save your work by clicking on the save button before you leave the editor!</p>
<strong>For developers and adventurers only:</strong>
<p>
You can edit the code of a template using exactly the same syntax as in the web interface. The code of the template
is in the template file located in the /etc/bibformat/format_templates/ directory of your CDS Invenio installation. You just
have to take care of the attributes of the template, which are saved in the same file as the code. See <a href="#attrsFormatTemplate">Edit the Attributes of a Format Template</a> to learn more about it.
</p>
<h4><a name="elementsInFormatTemplate">3.5 Use Format Elements</a></h4>
<p>To add a dynamic behaviour to your format templates, that is display for example a different title
for each record or a different background color depending on the type of record, you can use the format elements.</p>
<p>Format elements are the smart bricks you can copy-paste in your code to get the attributes of template
that change depending on the record. A format element looks like a regular HTML tag. </p>
<p>For example, to print
the title of a record, you can write <code><BFE_TITLE /></code> in your template code where you want to diplay the title</p>
<p>Format elements can take values as parameters. This allows to customize the behaviour of an element. For example you can write <code><BFE_TITLE prefix="Title: " /></code>, and BibFormat will take care of printing the title for you, with prefix "Title: ". The difference between <code>Title: <BFE_TITLE /></code> and <code><BFE_TITLE prefix="Title: " /></code> is that the first option will always write "Title: " while the second one will only print "Title: " if there exist a title for the record in the database. Of course there are chances that there is always a title for each record, but this can be useful for less common fields.</p>
<p>Some parameters are available for all elements. This is the case for the following ones:
<ul>
<li><code>prefix</code>: a prefix printed only if the record has a value for the element.</li>
<li><code>suffix</code>: a suffix printed only if the record has a value for the element.</li>
<li><code>default</code>: a default value printed if the record has no value for the element. In that case <code>prefix</code> and <code>suffix</code> are not printed.</li>
</ul>
</p>
<p>Some parameters are specific to elements. To get information on all available format elements you can read the <a href="bibformatadmin.py/format_elements_doc">Format Elements Documentation</a>, which is generated dynamically for all existing elements. it will show you what the element do and what parameters it can take.</p>
<p>While format elements looks like HTML tags, they differ in the followings ways from traditional ones:
<ul>
<li>A format element is a single tag: you cannot have <code><BFE_TITLE >some text<BFE_TITLE /></code> but only <code><BFE_TITLE /></code>.</li>
<li>The values of the parameters accept any characters, including < and >. The only limitation is that you cannot use the type of quotes that delimit that value: you can have for example <code><BFE_TITLE someParam="a lot of single quotes ' ' ' ' "/></code> or <code><BFE_TITLE someParam='a lot of double quotes " " " '/></code>, but not <code><BFE_TITLE someParam="a lot of same quotes as delimiter " " " "/></code>.</li>
<li>Format elements names always start with <code>BFE_</code>.</li>
<li>Format element can expand on multiple lines.</li>
</ul>
</p>
<p><b>Tips:</b> you can use the special element <code><BFE_FIELD tag="" /></code> to print the value
of any field of a record in your templates. This practice is however not
recommended because it would necessitate to revise all format
templates if you did change the meaning of the MARC code schema.</p>
<h4><a name="previewFormatTemplate">3.6 Preview a Format Template</a></h4>
<p>To preview a format template go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page and click on the format template you want to preview to open the template editor. The editor contains a preview panel at the bottom of the page.</p>
<p>Simply click on " Reload Preview" button to preview the template (you do not need to save the code before previewing).<br/>
Use the "Language" menu to preview the template in a given language</p>
<p>You can fill in the "Search Pattern" field to preview a specific record. The search pattern uses exactly the same
syntax as the one used in the web interface. The only difference with the regular search engine is that only the first matching record is shown.</p>
<strong>For developers and adventurers only:</strong>
<p>If you do not want to use the web interface to edit the templates but still would like to get previews, you can open the preview frame of any format in a new window/tab. In this mode you get a preview of the template (if it is placed in the /etc/bibformat/format_templates/ directory of your CDS Invenio installation). The parameters of the preview are specified in the url: <ul>
<li><code>bft</code>: the filename of the format template to preview</li>
<li><code>ln</code>: the language to use for the preview</li>
<li><code>pattern_for_preview</code>: the search pattern to use for the preview</li>
<p>You can add translations to your format templates. To do so enclose the text you want to localize
with tags corresponding to the two letters of the language. For example if we want to localize "title", write <code><en>Title</en></code>. Repeat this for each language in which you want to make "title" available: <code><en>Title</en><fr>Titre</fr><de>Titel</de></code>.
Finally enclose everything with <code><lang> </lang></code> tags: <code><lang><en>Title</en><fr>Titre</fr><de>Titel</de></lang></code></p>
<p>For each <lang> group only the text in the user's language is displayed. If user's language is not
available in the <lang> group, your default CDS Invenio language is used.</p>
<h4><a name="escapeFormatTemplate">3.8 Escaping special HTML/XML characters</a></h4>
<p>By default, BibFormat escapes all values returned by format
elements. As a format template designer, you can assume in almost all
cases that the values you get from a format element will be escaped
for you. For special cases, you can set the parameter
<code>escape</code> of the element to '0' when calling it, to make
BibFormat understand that it must not escape the values of the
element, or to '1' to force the escaping. </p>
<p>
For example
<code><bfe_abstract /></code> will return:<br/>
<code>[...]We find that for spatially-flat cosmologies, background lensing</code><br/>
<code>clusters with reasonable mass-to-light ratios lying in the</code><br/>
<code>redshift range 0<span style="color:red;">&lt;</span>1 are strongly excluded, [...]</code><br/><br/>
while <code><bfe_abstract escape="0"/></code> will return:<br/>
<code>[...]We find that for spatially-flat cosmologies, background lensing</code><br/>
<code>clusters with reasonable mass-to-light ratios lying in the</code><br/>
<code>redshift range 0<span style="color:red;"><</span>1 are strongly excluded, [...]</code><br/><br/>
</p>
<p>In most cases, you will not set <code>escape</code> to 1, nor 0, but
just let the developer of the element take care of that for you.</p>
<p>Please note that values given in special parameters
<code>prefix</code>, <code>suffix</code>, <code>default</code> and
<code>nbMax</code> are never escaped, whatever the value of
<code>escape</code> is (but other parameters will). You have to take
care of that in your format template, as well as of all other values that
are not returned by the format elements.</p>
<h3><a name="attrsFormatTemplate">3.9 Edit the Attributes of a Format Template</a></h4>
<p>To edit the attributes of a format template
go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page, click on
the format template you want to edit, and then in the menu click on "Modify Template Attributes".</p>
<p>
A format template contains two attributes:
<ul>
<li><code>Name</code>: the name of the template</li>
<li><code>Description</code>: a short description of the template</li>
</ul>
<p>Note that changing these parameters has no impact on the formatting. Their purpose in only to
document the template.</p>
<p>If the name you have chosen already exists for another template, you name will be suffixed with an integer so that the name is unique.</p>
<p>You should also be aware that if you change the name of a format template, all output formats that were linking to this template will be changed to match the new name.</p>
<strong>For developers and adventurers only:</strong>
<p>You can change the attributes of a template by editing its file in the /etc/bibformat/format_templates/ directory of your CDS Invenio installation. The attributes must be enclosed with tags <code><name> </name></code> and <code><description> </description></code> and should ideally be placed at the beginning of the file.</p>
<p>Also note that the admin web interface tries to keep the name of the template in sync with the filename of the template. If the name is changed through the web interface, the filename of the template is changed, and all output formats that use this template are updated. You have to do update output formats manually if you change the filename of the template without the web interface.</p>
<h3><a name="dependenciesFormatTemplate">3.10 Check the Dependencies of a Format Template</a></h3>
<p>To check the dependencies of a format template
go to the <a href="bibformatadmin.py/format_template_manage">Manage Format Template</a> page, click on
the format template you want to check, and then in the menu click on "Check Dependencies".</p>
<ul> <li>The output formats that use this format template</li>
<li>the elements used in the template (and Marc tags use in these elements in parentheses)</li>
<li>A summary of all the Marc tags involved in the elements of the template</li>
</ul>
Note that some Marc tags might be omitted.</p>
<h3><a name="validityFormatTemplate">3.11 Check the Validity a Format Template</a></h3>
<p>To check the validity of a format template, simply go to the <a href="bibformatadmin.py/format_templates_manage">Manage Format Templates</a> page, and look at the column 'status' for the format template you want to check. If message "Ok" is there,
then no problem was found with the template. If message 'Not Ok' is in the column, click on it to see
the problems that have been found for the template.</p>
<h3><a name="xslFormatTemplate">3.12 XSL Format Templates</a></h3>
<p>In addition to the HTML-like syntax introduced in previous sections, BibFormat also
has support for server-side XSL transformation. Although you can do all the formatting using this custom HTML syntax, there are
cases where an XSL stylesheet might be preferred. XSLT is for example a natural choice
when you need to output complex XML, especially when your XML has a deep tree structure.
You might also prefer using XSLT if you already feel comfortable with XSL syntax.</p>
<p>XSL format templates are written using regular XSL. The template file has to be placed in the same folder
as regular format template files, and its file extension must be <code>.xsl</code>. The XSL template
are also visible through the web interface, as any regular format template file. However, some
functions like the "Dependencies checker" or the possibility to create a template or edit its attributes are not
available for the XSL templates.</p>
<p>In BibFormat XSL you have access to the following functions, provided you have declared <code>xmlns:fn="http://cdsweb.cern.ch/bibformat/fn"</code> in your stylesheet:</p>
<dd>Returns the record modification date. Eg: <code><xsl:value-of select="fn:modification_date(445)"/></code> returns modification date of record 445</dd>
<dd>Returns the results of the evaluation of the format template code. Eg: <code><xsl:value-of select="fn:eval_bibformat(marc:controlfield[@tag='001'],'&lt;BFE_SERVER_INFO var=&quot;recurl&quot;>')" /></code> returns the url of the current record. The parameter <code><i>bibformat_template_code</i></code> is regular code used inside BibFormat format templates, with <code><</code> escaped as <code>&lt;</code> and <code>"</code>(quotes) escaped as <code>&quot;</code></dd>
</dl>
<p>Finally, please note that you will need to install a supported XSLT parser in order
to format using XSL stylesheets.</p>
<h2><a name="FormatElements">4. Format Elements</a></h2>
<p>Format elements are the bricks used in format templates to provide dynamic content to the formatting process.
Their purpose is to allow non computer literate persons to easily integrate data from the records in the database into their templates. </p>
<p>Format elements are typically written in Python (there is an exception to that point which is dicussed in <a href="#addFormatElement">Add a Format Element</a>). This brings great flexibily and power to the formatting process. This however restricts the creation of format elements to developers.</p>
<h3><a name="addFormatElement">4.1 Add a Format Element</a></h3>
<p>The most typical way of adding a format element is to drop a <code>.py</code> file in the lib/python/invenio/bibformat_elements directory of your CDS Invenio installation. See <a href="#codeFormatElement">Edit the Code of a Format Element</a> to learn how to implement an element.</p>
- <p>The most simple way to add a format element is to add a en entry in the "<a href="<WEBURL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module. When BibFormat cannot find the Python format element corresponding to a given name, it looks into this table for the name and prints the value of the field declared for this name. This lightweight way of doing is straightforward but does not allow complex handling of the data (it limits to printing the value of the field, or the values of the fields if multiple fields are declared under the same label).</p>
+ <p>The most simple way to add a format element is to add a en entry in the "<a href="<CFG_SITE_URL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module. When BibFormat cannot find the Python format element corresponding to a given name, it looks into this table for the name and prints the value of the field declared for this name. This lightweight way of doing is straightforward but does not allow complex handling of the data (it limits to printing the value of the field, or the values of the fields if multiple fields are declared under the same label).</p>
<h3><a name="removeFormatElement">4.2 Remove a Format Element</a></h3>
<p>To remove a Python format element simply remove the corresponding file from the lib/python/invenio/bibformat_elements directory of your CDS Invenio installation.</p>
- <p>To remove a format element declared in the "<a href="<WEBURL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module simply remove the entry from the table.</p>
+ <p>To remove a format element declared in the "<a href="<CFG_SITE_URL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module simply remove the entry from the table.</p>
<h3><a name="codeFormatElement">4.3 Edit the Code of a Format Element</a></h3>
- <p>This section only applies to Python format elements. Basic format elements declared in "<a href="<WEBURL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" have non configurable behaviour.</p>
+ <p>This section only applies to Python format elements. Basic format elements declared in "<a href="<CFG_SITE_URL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" have non configurable behaviour.</p>
<p>A format element file is like any regular Python program. It has to implement a <code>format</code> function, which returns a <code>string</code> and takes at least <code>bfo</code> as first parameter (but can take as many others as needed).</p>
<p>Here is for example the code of the "bfe_title.py" element:
<pre>
def format(bfo, separator=" "):
"""
Prints the title of a record.
@param separator separator between the different titles
"""
titles = []
title = bfo.field('245__a')
title_remainder = bfo.field('245__b')
titles.append( title + title_remainder )
title = bfo.field('0248_a')
if len(title) > 0:
titles.append( title )
title = bfo.field('246__a')
if len(title) > 0:
titles.append( title )
title = bfo.field('246_1a')
if len(title) > 0:
titles.append( title )
return separator.join(titles)
</pre>
In format templates this element can be called like a function, using HTML syntax:<br/>
Notice that the call uses (almost) the filename of your element. To find out which element to use, BibFormat tries different filenames until the element is found: it tries to <ol>
<li>ignore the letter case</li>
<li>replace underscore with spaces</li>
<li>remove the BFE_ from the name</li>
</ol>
This means that even if the filename of your element is "my element.py", BibFormat can resolve the call <BFE_MY_ELEMENT /> in a format template. This also means that you must take care no to have two format elements filenames that only differ in term of the above parameters.
</p>
<p>The <code>string</code> returned by the <code>format</code> function corresponds to the value that is printed instead of the format element name in the format template.</p>
<p>The <code>bfo</code> object taken as parameter by <code>format</code> stands for BibFormatObject: it is an object that represents the context in which the formatting takes place. For example it allows to retrieve the value of a given field for the record that is being formatted, or the language of the user. We see the details of the BibFormatObject further below.</p>
<p>The <code>format</code> function of an element can take other parameters, as well as default values for these parameters. The idea is that these parameters are accessible from the format template when calling the elements, and allow to parametrize the behaviour of the format element.</p>
<p>It is very important to document your element: this allows to generate a documentation for the elements accessible to people writing format templates. It is the only way for them to know what your element do. The key points are:
<ul>
<li>Provide a docstring for the <code>format</code> function</li>
<li>For each of the parameters of the <code>format</code> function (except for <code>bfo</code>), provide a description using a Java-like doc syntax in the doc string:<br/> <code>@param my_param description for my param</code> (one line per parameter)</li>
<li>You can use one <code>@see</code> followed by a comma separated list of elements filenames to provide a reference to other elements of interests related to this one: <br/> <code>@see my_element1.py, my element2.py</code></li>
</ul>
</p>
<p>Typically you will need to get access to some fields of a record to display as output. There are two ways to this: you can access the <code>bfo</code> object given as parameter and use the provided (basic) accessors, or import a dedicated module and use its advanced functionalities.</p>
<p><b>Method 1: Use accessors of <code>bfo</code></b>:<br/>
<code>bfo</code> is an instance of the <code>BibFormatObject</code> class. The following methods are available:
<ul>
<li><code>get_record()</code>: Returns the record of this BibFormatObject instance as a BibRecord structure. Allows advanced access on the structure using <code>BibRecord</code>.</li>
<li><code>control_field(tag)</code>: Returns the value of control field given by MARC <code>tag</code>.</li>
<li><code>field(tag)</code>:Returns the value of the field corresponding to MARC <code>tag</code>. If the value does not exist, return empty string.</li>
<li><code>fields(tag)</code>: Returns the list of values corresonding to MARC <code>tag</code>.If tag has an undefined subcode (such as 999C5), the function returns a list of dictionaries, whoose keys are the subcodes and the values are the values of tag.subcode. If the tag has a subcode, simply returns list of values corresponding to tag.</li>
<li><code>kb(kb, string, default="")</code>: Returns the value of the <code>string</code> in the knowledge base <code>kb</code>. If kb does not exist or string does not exist in kb, returns <code>default</code> string.</li>
</ul>
You can also get access to other information through <code>bfo</code>, such as the language in which the formatting should occur with <code>bfo.lang</code>. To learn more
-about the possibilities offered by the <code>bfo</code>, read the <a href="<WEBURL>/help/hacking/bibformat-api">BibFormat APIs</a>
+about the possibilities offered by the <code>bfo</code>, read the <a href="<CFG_SITE_URL>/help/hacking/bibformat-api">BibFormat APIs</a>
</p>
<p><b>Method 2: Use module <code>BibRecord</code></b>:<br/>
BibRecord is a module that provides advanced functionalities
regarding access to the field of a record
<code>bfo.get_record()</code> returns a structure that can be
understood by BibRecord's functions. Therefore you can import
the module's functions to get access to the fields you want.
</p>
<h3><a name="previewFormatElement">4.4 Preview a Format Element</a></h3>
<p>
You can play with a format element parameters and see the result
of the element directly in the
<a href="bibformatadmin.py/format_elements_doc">format elements documentation</a>:
for each element, under the section "See
also", click on "Test this element". You are redirected to a page
where you can enter a value for the parameters. A description is
associated with each parameter as well as an indication of the
default value of the parameter if you do not provide a custom
value. Click on the "Test!" button to see the result of the
<p>You can follow the standard internationalization procedure in
use accross CDS Invenio sources. For example the following code
will get you the translation for "Welcome" (assuming "Welcome" has
been translated):
<pre>
from invenio.messages import gettext_set_language
ln = bfo.ln
_ = gettext_set_language(ln)
translated_welcome = _("Welcome")
</pre>
</p>
<p>Notice the access to <code>bfo.ln</code> to get access to the
current language of the user. For simpler translations or
behaviour depending on the language you can simply check the value
<code>bfo.ln</code> to return your custom text.</p>
<h3><a name="escapeFormatElement">4.6 Escaping special HTML/XML characters</a></h3>
<p>In most cases, that is cases where your
element does not return HTML output, you do not have to take any
particular action in order to escape values that you output: the
BibFormat engine will take care of escaping the returned value of the element
for you. In cases where you want to return text that should not
be escaped (for example when you return HTML links), you can make
the formatting engine know that it should not escape your
value. This is done by implementing the
<code>escape_values(bfo)</code> function in your element, that
will return (int) 0 when escape should not be done (or 1 when
escaping should be done):
<pre>def escape_values(bfo):
"""
Called by BibFormat in order to check if output of this element
should be escaped.
"""
return 0
</pre>
Note that the function is given a <code>bfo</code> object as
parameter, such that you can do additional testing if your element
should really return 1 or 0 (for very special cases).<br/> Also note
that the behavior defined by the <code>escape_values()</code> function
will be overriden by the <code>escape</code> parameter used in the
format template if it is specified.
</p>
<p> Finally, be cautious when you disable escaping: you will have
to take care of escaping values "manually" in your format element
code, in order to avoid non valid outputs or XSS
vulnerabilities. This can be done easily when using the
<code>field</code>, <code>fields</code> and
<code>controlfield</code> functions of bfo with <code>escape</code>
parameter:
<pre>
title = bfo.field('245__a', escape="1")
abstract = bfo.field('520__a', escape="2")
</pre>
The <code>escape</code> parameter can be one of the following values:
<ul><li>0 - no escaping</li>
<li>1 - escape all HTML characters (escaped chars are shown as escaped)</li>
<li>2 - remove dangerous HTML tags to avoid XSS, but
keep basic one (such as <br />)
This is particularly useful if you want to store HTML text in your
metadata but still want to escape some tags to prevent
XSS vulnerabilities. Note that this method is slower than
basic escaping of mode 1.<br/>
Escaped tags are removed.</li>
<li>3 - mix of mode 1 and mode 2. If field_value starts with <!--HTML-->,
then use mode 2. Else use mode 1.</li>
<li>4 - remove all HTML/XML tags </li>
</ul>
These modes are the same for <code>escape_values(bfo)</code> function.
<p>
You can also decide not to use the <code>escape</code> parameter and escape values
using any other Python function/library you want to use (such as <code>cgi.escape()</code>).
</p>
<h3><a name="attrsFormatElement">4.7 Edit the Attributes of a Format Element</a></h3>
<p>A format element has mainly four kinds of attributes: <ul>
<li>Name: it corresponds to the filename of the element.</li>
<li>Description: the description is in the <code>docstring</code> of the <code>format</code> function (excepted lines prefixed with <code>@param</code> and <code>@see</code>).</li>
<li>Parameters descriptions: for each parameter of the <code>format</code> function, a line beginning with <code>@param</code> <i>parameter_name</i> and followed by the description of the parameter is present in the <code>docstring</code> of the <code>format</code> function.</li>
<li>Reference to other elements: one line beginning with <code>@see</code> and followed by a list of comma-separated format elements filenames in the in the <code>docstring</code> of the <code>format</code> function provides a link to related elements.</li>
</ul>
</p>
<h3><a name="dependenciesFormatElement">4.8 Check the Dependencies of a Format Element</a></h3>
<p>There are two ways to check the dependencies of a format element. The simplest way is to go to the <a href="bibformatadmin.py/format_elements_doc">format elements documentation</a> and click on "Dependencies of this element" for the element you want to check.</p>
<p>The second method to check the dependencies of an element is through regular unix tools: for example <code>$ grep -r -i 'bfe_<i>your_element_name</i>' .</code> inside the format templates directory will tell you which templates call your element.</p>
<h3><a name="validityFormatElement">4.9 Check the Validity of a Format Element</a></h3>
<p>There are two ways to check the validity of an element. The simplest one is to go to the <a href="bibformatadmin.py/format_elements_doc">format elements documentation</a> and click on "Correctness of this element" for the element you want to check.</p>
<p>The second method to check the validity of an element is through regular Python methods: you can for example import the element in the interactive interpreter and feed it with test parameters. Notice that you will need to build a BibFormatObject instance to pass as <code>bfo</code> parameter to the <code>format</code> function of your element.</p>
<h3><a name="browseDocFormatElement">4.10 Browse the Format Elements Documentation</a></h3>
<p>Go to the <a href="bibformatadmin.py/format_elements_doc">format elements documentation</a>. There is a summary of all available format elements at the top of the page. You can click on an element to go to its detailed description in the second part of the page.</p>
<p>Each detailed documentation shows you:
<ul>
<li>A description of what the element does.</li>
<li>A list of all parameters you can use for this element.</li>
<li>For each parameter, a description and the default value when parameter is ommitted.</li>
<li>A link to a tool to track the dependencies of your element.</li>
<li>A link to a tool to check the correctness of your element.</li>
<li>A link to a tool to test your element with custom parameters.</li>
</ul></p>
<h2><a name="KBs">5. Knowledge Bases</a></h2>
<p>Knowledge bases are a way to define easily extendable repositories of mappings. Their use is various, but their main purpose is to get, given a value, the normalized version of this value. For example you may use a knowledge base to hold a list of all ways to abbreviate the name of a journal, and map these abbreviations to the full journal name. This would be useful to get a normalized journal name accross all of your records.</p>
<p>The knowledge base itself offers no method to do this normalization. It is limited to the archiving of this knowledge. To benefit from the normalization you need to use a format element which is knowledge-base-aware. The element will look by iteself into the knowledge base to format a record. In that way you can extend the formatting capabilities of this element without having to modify it.</p>
<h3><a name="addKB">5.1 Add a Knowledge Base</a></h3>
<p>To add a knowledge base go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page.
At the bottom of the page click on the "Add New Knowledge Base" button. The knowledge base has been created and you are asked to fill in its attribute. See <a href="#attrsKB">Edit the Attributes of a Knowledge Base</a> to learn more about the attributes of knowledge bases.</p>
<h3><a name="removeKB">5.2 Remove a Knowledge Base</a></h3>
<p>To remove a knowledge base go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page. Click on the "Delete" button facing the knowledge base you want to remove and confim. The knowledge base and all the mapping it includes are removed.</p>
<h3><a name="addMappingKB">5.3 Add a Mapping</a></h3>
<p>Go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page and click on the knowledge base for which you want to add a mapping. Fill in the form of the "Add New Mapping" section on the left of the page with the new mapping, and click on "Add New Mapping". The mapping has been created. Alternatively you can create the mapping without its attributes, and fill them afterward (See <a href="#editMappingKB">Edit a Mapping</a>).<p/>
<h3><a name="removeMappingKB">5.4 Remove a Mapping</a></h3>
<p>Go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page and click on the knowledge base for which you want to remove a mapping. Click on the "Delete" button facing the mapping you want to delete.
<h3><a name="editMappingKB">5.5 Edit a Mapping</a></h3>
<p>Go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page and click on the knowledge base for which you want to edit a mapping. Locate the mapping in the list. You can click on the column headers to order the list by <i>Map From</i> or by <i>Map To</i> to help you find it. Once you have edited the mapping click on the corresponding "Save" button.
<h3><a name="attrsKB">5.6 Edit the Attributes of a Knowledge Base</a></h3>
Go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> administration page and click on the knowledge base you want to edit. In the top menu, click on "Knowledge Base Attributes". You can then give your knowledge base a name and a description. Finally click on the "Update Base Attributes" button.
<h3><a name="validityKB">5.7 Check the Dependencies a Knowledge Base</a></h3>
To check the dependencies of a knowledge base
go to the <a href="bibformatadmin.py/kb_manage">Manage Knowledge Bases</a> page, click on
the knowledge base you want to check, and then in the menu click on "Knowledge Base Dependencies".</p>
<p>The next page shows you the list of format elements that use this knowledge base.</p>
<p>Note that some format elements might be omitted.</p>
<h2><a name="BibReformat">6. Run BibReformat</a></h2>
<p>While records can be formatted on-the-fly using BibFormat, it is usually necessary to preformat the records
in order to decrease the load of your server. To do so, use the <code>bibreformat</code> command line tool.</p>
<h3><a name="runBibReformat">6.1 Run BibReformat</a></h3>
<p>The following options are available for running <code>bibreformat</code>:<p>
<blockquote>
<pre> Usage: bibreformat [options]
-u, --user=USER User name to submit the task as, password needed.
-s, --sleeptime=SLEEP Time after which to repeat tasks (no)
-t, --time=DATE Moment for the task to be active (now).
-a, --all All records
-c, --collection Select records by collection
-f, --field Select records by field.
-p, --pattern Select records by pattern.
-o, --format Specify output format to be (re-)created. (default HB)
-n, --noprocess Count records to be processed only (no processing done)
Example: bibreformat -n Show how many records are to be bibreformated.
</pre>
</blockquote>
For example, to reformat all records in HB (=HTML brief) format, you'd launch:
<blockquote>
<pre>$ bibreformat -a -oHB
</pre>
</blockquote>
and you watch the progress of the process via <code>bibsched</code>.
<p>Note that BibReformat understands <code>-p</code>, <code>-f</code>,
and <code>-c</code> arguments that enable you to easily reformat only
the records you need. For example, to reformat the Pictures
collection, launch:
</p><blockquote>
<p>
<pre>$ bibreformat -cPictures -oHB
</pre>
</blockquote>
or to reformat HD (=HTML detailed) format for records #10 to #20, you
launch:
<blockquote>
<pre>$ bibreformat -p"recid:10->20" -oHD
</pre>
</blockquote>
<p>Last but not least, if you launch bibreformat without arguments:
</p><blockquote>
<pre>$ bibreformat
</pre>
</blockquote>
</p>
it will process all the records that have been modified since the last
run of BibReformat, as well as all newly inputted records. This is
suitable for running BibReformat in a periodical daemon mode via
-BibSched. See our <a href="<WEBURL>/help/admin/howto-run">HOWTO Run
+BibSched. See our <a href="<CFG_SITE_URL>/help/admin/howto-run">HOWTO Run
Your CDS Invenio Installation</a> guide for more information.</p>
<h2><a name="Appendix">7. Appendix</a></h2>
<h3><a name="marcNotation">7.1 MARC Notation in Formats</a></h3>
<p>The notation for accessing fields of a record are quite flexible. You can use a syntax strict regarding MARC 21, but also
a shortcut syntax, or a syntax that can have a special meaning.</p>
<p>The MARC syntax is the following one:
<code>tag[indicator1][indicator2] [$ subfield]</code> where <code>tag</code> is 3 digits, <code>indicator1</code> and <code>indicator2</code> are 1 character each, and <code>subfield</code> is 1 letter.
</p>
<p>For example to get access to an abstract you can use the MARC notation <code>520 $a</code>. You can use this syntax in BibFormat. However you can also:
<ul>
<li>Omit any whitespace character (or use as many as you want)</li>
<li>Omit the <code>$</code> character (or use as many as you want)</li>
<li>Omit or use both indicators. You cannot specify only one indicator. If you need to use only one, use underscore <code>_</code> character for the other indicator.</li>
<li>Use percent '<code>%</code>' instead of any character to specify all ("don't care" or wildcard character) for that character. </li>
</ul></p>
<h3><a name="migration">7.2 Migrating from Previous BibFormat</a></h3>
<p>The new Python BibFormat formats are not backward compatible with the previous formats. New concepts and capabilities have been introduced and some have been dropped. If you have not modified the "Formats" or modified only a
little bit the "Behaviours" (or modified "Knowledge Bases"), then the transition will be painless and
automatic. Otherwise you will have to manually rewrite some of the
formats. This should however not be a big problem.</p>
<p>The first thing you should do is to read the <a href="#shortIntro">Five Minutes Introduction to BibFormat</a> to understand how the new BibFormat works. We also assume that you are familiar with the concepts of the old BibFormat. As the new formats separate the presentation from the business logic (i.e. the bindings to the database), it is not possible to automatically handle the translation. This is why you should at least be able to read and understand the formats that you want to migrate.</p>
<h4>Differences between old and new BibFormat</h4>
<p>
The most noticeable differences are:<br/>
<br/>
a) "Behaviours" have been renamed "Output formats".<br/>
b) "Formats" have been renamed "Format templates". They are now
written in HTML.<br/>
c) "User defined functions" have been dropped.<br/>
d) "Extraction rules" have been dropped.<br/>
e) "Link rules" have been dropped.<br/>
f) "File formats" have been dropped.<br/>
g) "Format elements" have been introduced. They are written in Python,
and can simulate c), d) and e).<br/>
h) Formats can be managed through web interface or through
human-readable config files.<br/>
i) Introduction of tools like validator and dependencies checker.<br/>
j) Better support for multi-language formatting.<br/>
</p>
<p>
Some of the advantages are:<br/>
<br/>
+ Management of formats is much clearer and easier (less concepts,
more tools).<br/>
+ Writing formats is easier to learn : less concepts
to learn, redesigned work-flow, use of existing well known and
well documented languages.<br/>
+ Editing formats is easier: You can use your preferred HTML editor such as
Emacs, Dreamweaver or Frontpage to modify templates, or any text
editor for output formats and format elements. You can also use the
simplified web administration interface.<br/>
+ Faster and more powerful templating system.<br/>
+ Separation of business logic (output formats, format elements)
and presentation layer (format templates). This makes the management
of formats simpler.<br/>
</p>
<p>
The disadvantages are:<br/>
<br/>
- No backward compatibility with old formats.<br/>
- Stricter separation of business logic and presentation layer:<br/>
no more use of statements such as if(), forall() inside templates,
and this requires more work to put logic inside format elements.<br/>
</p>
<h4>Migrating <i>behaviours</i> to <i>output formats</i></h4>
<p>Behaviours were previously stored in the database and did require to use the evaluation language to
provide the logic that choose which format to use for a record. They also let you enrich records
with some custom data. Now their use has been simplified and rectricted to equivalence tests on the value of a field
of the record to define the format template to use.</p>
<p>translates to the following output format (in textual configuration file):</p>
<p>
<code>
tag 980__a:<br/>
PICTURE --- Picture_HTML_brief.bft<br/>
default: Default_HTML_brief.bft<br/>
</code><p/>
<p>or visual representation through web interface:<br/>
-<img src="<WEBURL>/img/admin/bibformat-guide-bfo_hb_migrate.png" alt="Image representation of HB output format" />
+<img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfo_hb_migrate.png" alt="Image representation of HB output format" />
</p>
<h4>Migrating <i>formats</i> to <i>format templates</i> and <i>format elements</i></h4>
<p>The migration of formats is the most difficult part of the migration. You will need to separate the presentation code (HTML) from the business code (iterations, tests and calls to the database). Here are some tips on how you can do this:</p>
<ul>
<li>If you want to save the time of unescaping all HTML characters and understanding how the layout should look like, just go with your web browser to a formatted version of the format in your CDS Invenio installation, and copy the source of the web page. Identify the parts of the HTML code which are specific to the current record, and replace them with a call to the corresponding format element.</li>
<li>If you have made small modifications to the old default provided formats, we suggest that you use the new provided ones and modify them according to your needs.</li>
</ul>
<h4>Migrating <i>UDFs</i> and <i>Link rules</i></h4>
<p><i>User Defined Functions</i> and <i>Link rules</i> have been dropped in the new BibFormat. These concepts have no reasons to be as they can be fully implemented in the <i>format elements</i>. For example the <code>AUTHOR_SEARCH</code> link rule can directly be implemented in the <code>Authors.bfe</code> element.</p>
<p>As for the UDFs, most of them are directly built-in functions of Python. Whenever a special function as to be implemented, it can be defined in a regular Python file and used in any element.</p>
<h4>The Migration Kit</h4>
<p>The migration kit is only available in older versions of Invenio.<br/>
To enable the migration kit in this release, you need to copy the required files from an older release.
</p>
<h4><a name="runSideBySide">Run old and new formats side by side</a></h4>
<p>This was possible by default only in older versions of Invenio.<br/>
To enable this functionality in this release, you need to copy the required files from an older release.
</p>
<h3><a name="integrationDreamweaver">7.3 Integrating BibFormat into Dreamweaver MX</a></h3>
<p>BibFormat templates have been thought to be editable in custom HTML editors. We propose in this section
a way to extend one particular editor, Dreamweaver.</p>
<h4>Make Dreamweaver Recognize Format Elements in Layout View</h4>
<p>To make Dreamweaver understand the format elements and display an icon for each of them in the layout editor, you must
edit a Dreamweaver configuration file named <code>Tags.xml</code> located inside /Configuration/ThirdPartyTags directory
of your Dreamweaver installation folder. At the end of this file, copy-paste the following lines:
- Also copy this icon <img src="<WEBURL>/img/admin/bibformat-guide-bfe.gif" alt="bibformat.gif"/> in the same directory as <code>Tags.xml</code> (right-click on icon, or ctrl-click on one-button mouse, and "Save Image As..."). Make sure the downloaded image is named "<code>bibformat.gif</code>".
+ Also copy this icon <img src="<CFG_SITE_URL>/img/admin/bibformat-guide-bfe.gif" alt="bibformat.gif"/> in the same directory as <code>Tags.xml</code> (right-click on icon, or ctrl-click on one-button mouse, and "Save Image As..."). Make sure the downloaded image is named "<code>bibformat.gif</code>".
</p>
<p>Note that Dreamweaver might not recognize Format Elements when complex formatting is involved due to these elements.</p>
<h4>Add a Format Elements Floating Panel</h4>
<p>You can add a floating panel that will you to insert Format Elements in your document and read the documentation
of all available Format Elements.</p>
<p>The first step is to declare in which menu of Dreamweaver this floating panel is going to be available.
To do so, edit file "<code>Menu.xml</code>" located inside /Configuration/Menus of your Dreamweaver
application directory and copy-paste the following line in the menu you want
(typically inside tag <code>'menu'</code> with attribute <code>id='DWMenu_Window_Others')</code>:</p>
<p>Once this is done, you can <a href="bibformatadmin.py/download_dreamweaver_floater">download the floating palette</a> (if file opens in your browser instead of downloading, right-click on icon, or ctrl-click on one-button mouse, and "Save Target As...") and move the dowloaded file "<code>BibFormat_floater.html</code>" (do not rename it) into /Configuration/Floaters directory of your Dreamweaver application folder.</p>
<p>To use the BibFormat floating panel, open Dreamweaver, and choose <code>Window > Others > BibFormat Elements</code>.</p>
<p>Whenever a new version of the palette is available, you can skip the edition of file "<code>Menu.xml</code>" and just replace the old "<code>BibFormat_floater</code>" file with the new one.</p>
<h3><a name="faq">7.4 FAQ</a></h3>
<h4>Why do we need output formats? Wouldn't format templates be sufficient?</h4>
<p>As you potentially have a lot of records, it is not conceivable to specify for each of them which
format template they should use. This is why this rule-based decision layer has been introduced.</p>
<h4>How can I protect a format?</h4>
<p>As a web user, you cannot protect a format. If you are administrator of the
system and have access to the format files, you can simply use the permission rights of your system, as BibFormat
is aware of it.</p>
<h4>Why cannot I edit/delete a format?</h4>
<p>The format file has certainly been protected by the administrator of the server. You must ask the
administrator to unprotect the file if you want to edit it.</p>
<h4>How can I add a format element from the web interface?</h4>
<p>Format elements cannot be added, removed or edited through the web interface. This limitation
-has been introduced to limit the security risks caused by the upload of Pythonic files on the server. The only possibility to add a basic format element from the web interface is to add a en entry in the "<a href="<WEBURL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module (see <a href="#addFormatElement">Add a Format Element</a>)</p>
+has been introduced to limit the security risks caused by the upload of Pythonic files on the server. The only possibility to add a basic format element from the web interface is to add a en entry in the "<a href="<CFG_SITE_URL>//admin/bibindex/bibindexadmin.py/field">Logical Fields</a>" management interface of the BibIndex module (see <a href="#addFormatElement">Add a Format Element</a>)</p>
<h4>Why are some Marc codes omitted in the "Check Dependencies" pages?</h4>
<p>When you check the dependencies of a format, the page reminds you that
some use of Marc codes might not be indicated. This is because it is not
possible (or at least not trivial) to guess that the call to <code>field(str(5+4)+"80"+"__a")</code>
is equal to a call to <code>field("980__a")</code>. You should then not completely rely on this indication.</p>
<h4>How are displayed deleted record?</h4>
<p>By default, CDS Invenio displays a standard "The record has been deleted." message for all
output formats with a 'text/html' content type. Your output format, format templates and format elements
are bypassed by the engine.
However, for more advanced output formats, CDS Invenio
goes through the regular formatting process and let your formats do the job. This allows you to customize how a record should be displayed once it has been deleted.</p>
<h4>Why are some format elements omitted in the "Knowledge Base Dependencies" page?</h4>
<p>When you check the dependencies of a knowledge base, the page
reminds you that format elements using this knowledge base might not
be indicated. This is because it is not possible (or at least not
trivial) to guess that the call to
<code>kb(e.upper()+"journal"+"s")</code> in a format element is equal
to a call to <code>kb("Ejournals")</code>. You should then not
completely rely on this indication.</p>
<h4>Why are some format elements defined in field table omitted in the format element documentation?</h4>
<p>Some format elements defined in the "Logical Fields" management
interface of the BibIndex module (the basic format elements) are not
shown in the format elements documentation pages. We do not show such
an element if its name starts with a number. This is to reduce the
number of elements shown in the documentation as the logical fields
table contains a lot of not so useful fields to be used in
templates.</p>
<h4>How can I get access to repeatable subfields from inside a format element?</h4>
<p>Given that repeatable subfields are not frequent, the
<code>bfo.fields(..)</code> function has been implemented to return the most convenient structure for most cases, that is a '<em>list of strings</em>' (<b>Case 1</b> below) or '<em>list of dict of strings</em>' (<b>Case 2</b> below). For eg. with the following metadata:
<description>Output the various actions available on a record</description>
<ul class="detailedrecordactions">
- <li><a href="<BFE_SERVER_INFO var='weburl'/>yourbaskets/add?ln=<BFE_SERVER_INFO var='lang'/>&recid=<BFE_RECORD_ID/>">_(Add to personal basket)_</a></li>
+ <li><a href="<BFE_SERVER_INFO var='CFG_SITE_URL'/>yourbaskets/add?ln=<BFE_SERVER_INFO var='lang'/>&recid=<BFE_RECORD_ID/>">_(Add to personal basket)_</a></li>
self.record_74_hd_title = '''<center><big><big><strong>Quasinormal modes of Reissner-Nordstrom Anti-de Sitter Black Holes</strong></big></big></center>'''
self.record_74_hd_abstract = '''<small><strong>Abstract: </strong>Complex frequencies associated with quasinormal modes for large Reissner-Nordstr$\ddot{o}$m Anti-de Sitter black holes have been computed. These frequencies have close relation to the black hole charge and do not linearly scale withthe black hole temperature as in Schwarzschild Anti-de Sitter case. In terms of AdS/CFT correspondence, we found that the bigger the black hole charge is, the quicker for the approach to thermal equilibrium in the CFT. The propertiesof quasinormal modes for $l>0$ have also been studied.</small><br />'''
Caption</span><br /> <small>Conference "Internet, Web, What's next?" on 26 June 1998 at CERN : Tim Berners-Lee, inventor of the World-Wide Web and Director of the W3C, explains how the Web came to be and give his views on the future.</small></p><p><span class="blocknote">
Légende</span><br /><small>Conference "Internet, Web, What's next?" le 26 juin 1998 au CERN: Tim Berners-Lee, inventeur du World-Wide Web et directeur du W3C, explique comment le Web est ne, et donne ses opinions sur l'avenir.</small></p>'''
<abstract>In its Euclidean formulation, the AdS/CFT correspondence begins as a study of Yang-Mills conformal field theories on the sphere, S^4. It has been successfully extended, however, to S^1 X S^3 and to the torus T^4. It is natural tohope that it can be made to work for any manifold on which it is possible to define a stable Yang-Mills conformal field theory. We consider a possible classification of such manifolds, and show how to deal with the most obviousobjection : the existence of manifolds which cannot be represented as boundaries. We confirm Witten's suggestion that this can be done with the help of a brane in the bulk.</abstract>
</front>
<article-type>research-article</article-type>
<ref/>
</article>
-</articles>''' % {'weburl': weburl}
+</articles>''' % {'siteurl': CFG_SITE_URL}
def test_nlm_output(self):
"""bibformat - NLM output"""
- pageurl = weburl + '/record/70?of=xn'
+ pageurl = CFG_SITE_URL + '/record/70?of=xn'
result = test_web_page_content(pageurl,
expected_text=self.record_70_xn)
self.assertEqual([], result)
class BibFormatBriefHTMLTest(unittest.TestCase):
"""Check output produced by BibFormat for brief HTML ouput for
#Offer the possibility to make a duplicate of existing format template code
out += '''
<table><tr>
- <th class="adminheaderleft">Make a copy of format template: [<a href="%(weburl)s/help/admin/bibformat-admin-guide#addFormatTemplate">?</a>]</th>
+ <th class="adminheaderleft">Make a copy of format template: [<a href="%(siteurl)s/help/admin/bibformat-admin-guide#addFormatTemplate">?</a>]</th>
out +='''<td>2. <small><a href="format_template_show_attributes?ln=%(ln)s&bft=%(filename)s">%(modify_template_attributes)s</a></small> </td>
<p>From here you can add, edit or delete output formats available for collections. Output formats define which template to use. <br/>To edit templates go to the <a href="format_templates_manage?ln=%(ln)s">template administration page</a>.</p>
<td><input tabindex="1" name="visibility" type="checkbox" id="outputFormatVisibility" %(visibility)s %(disabled)s value="1" /><small><label for="outputFormatVisibility">Show in list of available output formats (on public pages)</label></small></td>
<td><input tabindex="2" name="content_type" type="text" id="outputFormatContentType" size="25" value="%(content_type)s" %(readonly)s/> <small>Mime content-type. Specifies how the browser should handle this output.</small></td>
_ = gettext_set_language(ln) # load the right message language
out = '''
- <p>To add a new basic element (only fetch the value of a field, without special post-processing), go to the <a href="%(weburl)sadmin/bibindex/bibindexadmin.py/field">BibEdit "Manage Logical Fields"</a> page and add a name for a field. Make sure that the name is unique and corresponds well to the field. For example, to add an element that fetch the value of field 245__%, add a new logical field with name "title" and field "245__%". Then in your template, call BFE_TITLE to print the title.</p>
+ <p>To add a new basic element (only fetch the value of a field, without special post-processing), go to the <a href="%(siteurl)sadmin/bibindex/bibindexadmin.py/field">BibEdit "Manage Logical Fields"</a> page and add a name for a field. Make sure that the name is unique and corresponds well to the field. For example, to add an element that fetch the value of field 245__%, add a new logical field with name "title" and field "245__%". Then in your template, call BFE_TITLE to print the title.</p>
<p>To add a new complex element (for eg. special formatting of the field, condition on the value, etc.) you must go to the lib/python/invenio/bibformat_elements directory of your Invenio installation, and add a new format element file. Read documentation for more information.</p>
- ''' % {'weburl':weburl}
+ ''' % {'siteurl':CFG_SITE_URL}
return out
def tmpl_admin_kbs_management(self, ln, kbs):
"""
Returns the main management console for knowledge bases.
@param ln language
@param kbs a list of dictionaries with knowledge bases attributes
@return main management console as html
"""
_ = gettext_set_language(ln) # load the right message language
# FIXME: what about admin_email? needed? or use new arguments like CFG_SITE_ADMIN_EMAIL?
def format(bfo, var=''):
'''
Print several server specific variables.
- @param var the name of the desired variable. Can be one of: name, i18n_name, lang, CFG_VERSION, admin_email, support_email, weburl, searchurl, recurl
+ @param var the name of the desired variable. Can be one of: name, i18n_name, lang, CFG_VERSION, admin_email, support_email, CFG_SITE_URL, searchurl, recurl
name: the name of the server
i18n_name: internationalized name
lang: the default language of the server
CFG_VERSION: the software version
admin_email: the admin email
support_email: the support email
- weburl: the base url for the server
+ CFG_SITE_URL: the base url for the server
searchurl: the search url for the server
recurl: the base url for the record
'''
recID = bfo.recID
if var == '':
out = ''
elif var == 'name':
out = CFG_SITE_NAME
elif var == 'i18n_name':
out = CFG_SITE_NAME_INTL.get(bfo.lang, CFG_SITE_NAME)
3.1 <a href="#3.1">Definition of OAI sets</a><br />
3.2 <a href="#3.2">Exposing metadata via OAI Repository Gateway</a><br />
<a name="1"></a><h2>1. Overview</h2>
The BibHarvest module handles metadata gathering and delivery between OAI-PMH v.2.0 compliant repositories. Metadata exchange is performed on top of the OAI-PMH, the Open Archives Initiative's Protocol for Metadata Harvesting.
The BibHarvest Admin Interface can be used to set up a database of OAI sources and determine the periodicity of harvesting.
<p>In order to periodically harvest metadata from one or several repositories, it is possible to organize OAI sources through the BibHarvest Admin Interface.
The interface allows the administrator to add new repositories as well as edit and delete existing ones.
Once the database has been set up thorugh the BibHarvest Admin Interface, run the oaiharvest command-line tool to run periodical harvesting</p>
<p>The first step requires the administrator to enter the baseURL of the OAI repository. This is done for validation purposes - i.e. to check that the baseURL actually points to an OAI-compliant repository.
(Note: the validation simply performs an 'Identify' query to the baseURL and parses the reply with crucial tags such as <code>OAI-PMH</code> and <code>Identify</code>).
<br />Once the baseURL is validated, the administrator is required to fill into the following fields:
<br /><ul>
<li><strong>Source name</strong>: this will be the (alphanumeric) name used to refer to this OAI repository.</li>
<li><strong>Metadata prefix</strong>: upon validation the baseURL is checked for the available metadata formats. The administator can choose the desired format to harvest metadata in.</li>
<li><strong>Sets</strong>: upon validation the baseURL is checked for the available sets of the repository. The administator can choose the desired sets to harvest. If none is checked, all records are harvested. Otherwise selective harvesting for checked sets is done.</li>
<li><strong>Frequency</strong>: how often do you intend to harvest from this repository? (Note: selecting "never" excludes this source from automatic periodical harvesting)</li>
<li><strong>Starting date</strong>: on your first harvesting session, do you intend to harvest the whole repository (from beginning) or only newly added material (from today)? (WARNING: harvesting large collections of material may take very long!)</li>
<li><strong>Postprocess</strong>: how is the harvested material be dealt with after harvesting?
<ul><li>h: harvest only - metadata will be gathered and saved locally<li>h-u: harvest and upload - metadata will be automatically queued for upload (this is useful when harvested metadata is already in marcxml format)</li><li>h-c: harvest and convert - harvested metadata will be converted and saved locally</li><li>h-c-u: harvest, convert and upload - harvested metadata will be converted and queued for upload - this is useful when harvested metadata is in a format different from marcxml (e.g. oai_dc)</li><li>h-c-f-u: harvest, convert, filter, upload - useful when you want to filter harvested data in some non-trivial way before uploading</li></ul></li>
<li><strong>BibConvert configuration file</strong>: if postprocess involves conversion, the filename of an existing BibConvert configuration file (or its full path if file is not in standard BibConvert config directory) is needed.
This file determines how the metadata conversion will take place (e.g. oai_dc to marcxml).
In order to ensure that the configuration file performs the desired conversion, please test it in advance and follow the instruction contained in the <a href="../bibconvert/guide.html">BibConvert Admin Guide</a>.
Please note that some conversion parameters previously input at the command line can now be handled by the BibConvert Configuration Language (namely record separator, file header and file footer). Please consult the <a href="../bibconvert/guide.html">BibConvert Admin Guide</a> for more information on this.
This allows fully automatised harvesting and conversion (and possibly upload) of records provided a valid, functioning, BibConvert template is provided.</li>
</ul>
<b>Edit and delete OAI sources</b><br />
<p>Once a source has been added to the database, it will be visible in the overview page, as shown below</p>
<p>At this point it will be possible to edit the definition of this source by clicking on the appropriate action button. All the fields described in 2.2.1 can be modified (except for the Starting date).
There is no more validation at this stage, hence, please take extra care when editing important fields such as baseurl and metadataprefix.
<br />
OAI repositories can be removed from the database by clicking on the appropriate action button in the overview page.
<p>Once administrators have set up their desired OAI repositories in the database through the Admin Interface they can invoke <code>oaiharvest</code> to start up periodical harvesting.<br />
<br />
<b>Oaiharvest usage</b>
<blockquote>
<pre>
oaiharvest [options]
Specific options:
-r, --repository=REPOS_ONE,"REPOS TWO" name of the OAI repositories to be harvested (default=all)
-d, --dates=yyyy-mm-dd:yyyy-mm-dd harvest repositories between specified dates (overrides repositories' last updated timestamps)
Scheduling options:
-u, --user=USER user name to store task, password needed
-s, --sleeptime=SLEEP time after which to repeat tasks (no)
e.g.: 1s, 30m, 24h, 7d
-t, --time=TIME moment for the task to be active (now)
e.g.: +15s, 5m, 3h , 2002-10-27 13:57:26
General options:
-h, --help print this help and exit
-V, --version print version and exit
-v, --verbose=LEVEL verbose level (from 0 to 9, default 1)
</pre>
</blockquote>
<code>oaiharvest</code> performs a number of operations on the repositories listed in the database. By default <code>oaiharvest</code> considers all repositories, one by one (this gets overridden when <code>--repository</code> argument is passed).
<br /> For each repository that is considered, <code>oaiharvest</code> behaves according to the arguments passed at the command line:
<ul>
<li> If the <code>--dates</code> argument is not passed, it checks whether an update from the repository is needed (Note: the update status is calculated based on the time of the last harvesting and the frequency chosen by the administrator).
<ul>
<li>If an update is needed, it launches <code>bibharvest</code> and harvests all the metadata that the repository has added since the data of the last update. When the update is finished, the last update value is set to the current time and date.
<li>If an update is not needed, the source is skipped.
</ul>
<li> If the <code>--dates</code> argument is passed, it simply harvests the metadata of the repository from/until the given dates. The last update date is left unchanged.
<li> Finally, it performs the operations indicated in the postprocess mode, i.e. convert and/or upload the harvested metadata.
</ul>
<br />
<b>Oaiharvest usage examples</b><br />
<p>In most cases, administrators will want <code>oaiharvest</code> to run in the background, i.e. run in sleep mode and wake up periodically (e.g. every 24 hours) to check whether updates are needed, e.g. <code>oaiharvest -s 24h</code><br />
<p>In other cases, administrators may want to perform periodical harvesting only on specific sources, e.g. <code>oaiharvest -r cdsweb -s 12h</code><br />
<p>Another option is that administrators may want to harvest from certain repositories within two specific dates. This will be regarded as a one-off operation and will not affect the last update value of the source, e.g. <code>oaiharvest -r cdsweb -d 2005-05-05:2005-05-30</code><br />
<br />
<a name="3"></a><h2>3. OAI Repository</h2>
The OAI Repository corresponds to a set of metadata exposed for periodical harvesting by external OAI service providers. The following steps have to be done in order to expose metadata via OAI:
<ul>
<li>Definition of OAI sets</li>
<li>Exposing metadata via OAI Repository Gateway</li>
</ul>
<a name="3.1"></a><h3>3.1. Definition of OAI sets</h3>
The OAI repository is composed of OAI sets that correspond to a part of your metadata base. OAI sets can be configured via the <a href="oaiarchiveadmin.py">OAI Repository Admin Interface</a>, featuring essential OAI repository maintenance tasks.
<p>Each OAI set is defined by one or a union of several definitions. Each such definition is provided separately and corresponds to one line as displayed in the OAI Repository Admin overview page. The following information displays:
<br /><ul>
<li><strong>id: </strong>OAI set definition identifier (unique).<br /><br />
<li><strong>setSpec: </strong>OAI set identifier; OAI set is defined as a union of definitions with the same setSpec.<br /><br /><li><strong>setName: </strong>OAI set name; a human readable label<br /><br />
<li><strong>setCollection: </strong>Restriction to a particular data collection<br /><br />
<li><strong>p(N): </strong>phrase N; phrases are combined by logical AND<br /><br />
<li><strong>f(N): </strong>field that corresponds to phrase N<br /><br />
<li><strong>m(N): </strong>syntactic mode of phrase N<br /><br />
</ul>
<p>
<a name="3.2"></a><h3>3.2. Exposing metadata via OAI Repository Gateway</h3>
<p>Once OAI Repository is defined, the next step is to expose corresponding metadata via the OAI Repository Gateway. This is done by launching the <code>oaiarchive</code> script.<br />
<br />
<b>Oaiarchive usage</b>
<blockquote>
<pre>
oaiarchive [options]
Options:
-o --oaiset= Specify setSpec
-h --help Print this help
-V --version Print version information and exit
Modes
-a --add Add records to OAI repository
-d --delete Remove records from OAI repository
-r --report Print OAI repository status
-i --info Give info about OAI set (default)
Additional parameters:
-p --upload Upload records
-u --user=USER User name to submit the task as, password needed.
-s --sleeptime=SLEEP Time after which to repeat tasks (no)
-t --time=DATE Moment for the task to be active (now).
</pre>
</blockquote>
To expose set 'setname' via OAI repository gateway:
<blockquote>
<pre>
oaiarchive -apo 'setname' -s24
</pre>
</blockquote>
To remove records defined by set 'setname' from OAI repository:
<blockquote>
<pre>
oaiarchive -dpo 'setname'
</pre>
</blockquote>
To print OAI set status launch:
<blockquote>
<pre>
oaiarchive -io 'setname'
</pre>
</blockquote>
To print out the current status of the OAI repository launch:
<blockquote>
<pre>
oaiarchive -r
</pre>
</blockquote>
<p>
Please note that the <code>oaiarchive</code> script can be scheduled via <code>BibSched</code> in order to periodically update the OAI Repository with respect to database modifications and OAI set definitions modifications.
<p>
Please see also invenio.conf for more fine configurations of the OAI Repository.
In order to expose all public records (the entire content of the Home
collection) via the OAI Repository gateway, a predefined
<code>global</code> set can be used. To expose the global set with
periodical updates on daily basis launch:
<blockquote>
<pre>
$ oaiarchive -apo global -s24h
</pre>
</blockquote>
To perform a reverse operation, i.e. to remove all records from the global OAI set, remove the oaiarchive task from the <code>BibSched</code> queue and launch:
for (oai_src_id,oai_src_name,oai_src_baseurl,oai_src_prefix,oai_src_frequency,oai_src_config,oai_src_post,oai_src_bibfilter,oai_src_setspecs) in oai_src:
text = bibharvest_templates.tmpl_print_brs(CFG_SITE_LANG, 1)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "Source name", name = "oai_src_name", value = oai_src_name)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "Base URL", name = "oai_src_baseurl", value = oai_src_baseurl)
sets = findSets(oai_src_baseurl)
if sets:
# Show available sets to users
sets_specs = [set[0] for set in sets]
sets_names = [set[1] for set in sets]
sets_labels = [((set[1] and set[0]+' ('+set[1]+')') or set[0]) \
for set in sets]
sets_states = [ ((set[0] in oai_src_sets and 1) or 0) for set in sets]
text += bibharvest_templates.tmpl_admin_checkboxes(ln=CFG_SITE_LANG,
title="Sets",
name="oai_src_sets",
values=sets_specs,
labels=sets_labels,
states=sets_states)
else:
# Let user specify sets in free textbox
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG,
title = "Sets",
name='oai_src_sets',
value=' '.join(oai_src_sets))
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "Metadata prefix", name = "oai_src_prefix", value = oai_src_prefix)
text += bibharvest_templates.tmpl_admin_w200_select(ln = CFG_SITE_LANG, title = "Frequency", name = "oai_src_frequency", valuenil = "- select frequency -" , values = freqs, lastval = oai_src_frequency)
text += bibharvest_templates.tmpl_admin_w200_select(ln = CFG_SITE_LANG, title = "Postprocess", name = "oai_src_post", valuenil = "- select mode -" , values = posts, lastval = oai_src_post)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "BibConvert configuration file (if needed by postprocess)", name = "oai_src_config", value = oai_src_config)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "BibFilter program (if needed by postprocess)", name = "oai_src_bibfilter", value = oai_src_bibfilter)
text += bibharvest_templates.tmpl_print_brs(CFG_SITE_LANG, 2)
output += createhiddenform(action="editsource#1",
text=text,
button="Modify",
oai_src_id=oai_src_id,
ln=ln,
confirm=1)
if confirm in [1, "1"] and not oai_src_name:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a name for the source.")
elif confirm in [1, "1"] and not oai_src_prefix:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a metadata prefix.")
elif confirm in [1, "1"] and not oai_src_baseurl:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a base url.")
elif confirm in [1, "1"] and not oai_src_frequency:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please choose a frequency of harvesting")
elif confirm in [1, "1"] and not oai_src_post:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please choose a postprocess mode")
elif confirm in [1, "1"] and oai_src_post.startswith("h-c") and (not oai_src_config or validatefile(oai_src_config)!=0):
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "You selected a postprocess mode which involves conversion.")
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a valid name of or a full path to a BibConvert config file or change postprocess mode.")
text = bibharvest_templates.tmpl_admin_w200_text(ln=CFG_SITE_LANG,
title="Source name",
name="oai_src_name",
value=oai_src_name)
metadatas = findMetadataFormats(oai_src_baseurl)
if metadatas:
# Show available metadata to user
prefixes = []
for value in metadatas:
prefixes.append([value, str(value)])
text += bibharvest_templates.tmpl_admin_w200_select(ln=CFG_SITE_LANG,
title="Metadata prefix",
name="oai_src_prefix",
valuenil="- select prefix -" ,
values=prefixes,
lastval=oai_src_prefix)
else:
# Let user specify prefix in free textbox
text += bibharvest_templates.tmpl_admin_w200_text(ln=CFG_SITE_LANG,
title="Metadata prefix",
name="oai_src_prefix",
value=oai_src_prefix)
sets = findSets(oai_src_baseurl)
if sets:
# Show available sets to users
sets_specs = [set[0] for set in sets]
sets_names = [set[1] for set in sets]
sets_labels = [((set[1] and set[0]+' ('+set[1]+')') or set[0]) \
for set in sets]
sets_states = [ ((set[0] in oai_src_sets and 1) or 0) \
for set in sets]
text += bibharvest_templates.tmpl_admin_checkboxes(ln=CFG_SITE_LANG,
title="Sets",
name="oai_src_sets",
values=sets_specs,
labels=sets_labels,
states=sets_states)
else:
# Let user specify sets in free textbox
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG,
title = "Sets",
name='oai_src_sets',
value=' '.join(oai_src_sets))
text += bibharvest_templates.tmpl_admin_w200_select(ln = CFG_SITE_LANG, title = "Frequency", name = "oai_src_frequency", valuenil = "- select frequency -" , values = freqs, lastval = oai_src_frequency)
text += bibharvest_templates.tmpl_admin_w200_select(ln = CFG_SITE_LANG, title = "Starting date", name = "oai_src_lastrun", valuenil = "- select a date -" , values = dates, lastval = oai_src_lastrun)
text += bibharvest_templates.tmpl_admin_w200_select(ln = CFG_SITE_LANG, title = "Postprocess", name = "oai_src_post", valuenil = "- select mode -" , values = posts, lastval = oai_src_post)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "BibConvert configuration file (if needed by postprocess)", name = "oai_src_config", value = oai_src_config)
text += bibharvest_templates.tmpl_admin_w200_text(ln = CFG_SITE_LANG, title = "BibFilter program (if needed by postprocess)", name = "oai_src_bibfilter", value = oai_src_bibfilter)
text += bibharvest_templates.tmpl_print_brs(CFG_SITE_LANG, 2)
output += createhiddenform(action="addsource#1",
text=text,
button="Add OAI Source",
oai_src_baseurl=oai_src_baseurl,
ln=ln,
confirm=1)
elif confirm in ["0", 0] and validate(oai_src_baseurl)[0] > 0:
- output += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, weburl = weburl, funcurl = "admin/bibharvest/bibharvestadmin.py/addsource", title = "Try again with another url", args = [])
+ output += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, funcurl = "admin/bibharvest/bibharvestadmin.py/addsource", title = "Try again with another url", args = [])
- output += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, weburl = weburl, funcurl = "admin/bibharvest/bibharvestadmin.py/index", title = "Go back to the OAI sources overview", args = lnargs)
+ output += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, funcurl = "admin/bibharvest/bibharvestadmin.py/index", title = "Go back to the OAI sources overview", args = lnargs)
if confirm in [1, "1"] and not oai_src_name:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a name for the source.")
if confirm in [1, "1"] and not oai_src_prefix:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a metadata prefix.")
if confirm in [1, "1"] and not oai_src_frequency:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please choose a frequency of harvesting")
if confirm in [1, "1"] and not oai_src_lastrun:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please choose the harvesting starting date")
if confirm in [1, "1"] and not oai_src_post:
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please choose a postprocess mode")
if confirm in [1, "1"] and oai_src_post.startswith("h-c") and (not oai_src_config or validatefile(oai_src_config)!=0):
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "You selected a postprocess mode which involves conversion.")
output += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "Please enter a valid name of or a full path to a BibConvert config file or change postprocess mode.")
"Encode special chars in string for XML-compliancy."
if strxml is None:
return strxml
else:
strxml = strxml.replace('&', '&')
strxml = strxml.replace('<', '<')
return strxml
def escape_space(strxml):
"Encode special chars in string for URL-compliancy."
strxml = strxml.replace(' ', '%20')
return strxml
def encode_for_url(strxml):
"Encode special chars in string for URL-compliancy."
strxml = strxml.replace('%', '%25')
strxml = strxml.replace(' ', '%20')
strxml = strxml.replace('?', '%3F')
strxml = strxml.replace('#', '%23')
strxml = strxml.replace('=', '%3D')
strxml = strxml.replace('&', '%26')
strxml = strxml.replace('/', '%2F')
strxml = strxml.replace(':', '%3A')
strxml = strxml.replace(';', '%3B')
strxml = strxml.replace('+', '%2B')
return strxml
def oai_header(args, verb):
"Print OAI header"
out = ""
out = out + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + "\n"
out = out + "<OAI-PMH xmlns=\"http://www.openarchives.org/OAI/2.0/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd\">\n"
out = out + " <responseDate>" + oaigetresponsedate() + "</responseDate>\n"
if verb:
out = out + " <request verb=\"%s\">%s</request>\n" % (verb, oaigetrequesturl(args))
out = out + " <%s>\n" % verb
else:
out = out + " <request>%s</request>\n" % (oaigetrequesturl(args))
return out
def oai_footer(verb):
"Print OAI footer"
out = ""
if verb:
out = "%s </%s>\n" % (out, verb)
out = out + "</OAI-PMH>\n"
return out
def oai_error_header(args, verb):
"Print OAI header"
out = ""
### out = "Content-Type: text/xml\n\n"
out = out + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + "\n"
out = out + "<OAI-PMH xmlns=\"http://www.openarchives.org/OAI/2.0/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd\">\n"
out = out + " <responseDate>" + oaigetresponsedate() + "</responseDate>\n"
out = out + " <request verb=\"%s\">%s</request>\n" % (verb, oaigetrequesturl(args))
return out
def oai_error_footer(verb):
"Print OAI footer"
out = verb
out = "</OAI-PMH>\n"
return out
def get_field(sysno, field):
"Gets list of field 'field' for the record with 'sysno' system number."
out = []
digit = field[0:2]
bibbx = "bib%sx" % digit
bibx = "bibrec_bib%sx" % digit
query = "SELECT bx.value FROM %s AS bx, %s AS bibx WHERE bibx.id_bibrec='%s' AND bx.id=bibx.id_bibxxx AND bx.tag='%s'" % (bibbx, bibx, sysno, field)
"Returns the first database BIB ID for the OAI identifier 'identifier', if it exists."
sysno = None
if identifier:
query = "SELECT DISTINCT(bb.id_bibrec) FROM bib%sx AS bx, bibrec_bib%sx AS bb WHERE bx.tag='%s' AND bb.id_bibxxx=bx.id AND bx.value='%s'" % (CFG_OAI_ID_FIELD[0:2], CFG_OAI_ID_FIELD[0:2], CFG_OAI_ID_FIELD, identifier)
out += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG,
"OAI set definition %s added." % oai_set_name)
out += "<br />"
lnargs = [["ln", ln]]
out += "<br /><br />"
- out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, weburl = weburl, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs)
+ out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs)
out += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG,
"OAI set definition #%s edited." % oai_set_id)
out += "<br />"
lnargs = [["ln", ln]]
out += "<br />"
- out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, weburl = weburl, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs)
+ out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs)
question = """Do you want to delete the OAI definition #%s?""" % oai_set_id
text = bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, question)
text += "<br /><br /><br />"
text += pagebody_text("%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s-%s" % (oai_set_spec,
oai_set_name,
oai_set_collection,
oai_set_p1,
oai_set_f1,
oai_set_m1,
oai_set_op1,
oai_set_p2,
oai_set_f2,
oai_set_m2,
oai_set_op2,
oai_set_p3,
oai_set_f3,
oai_set_m3))
out += createform(action="delset",
text=text,
button="Delete",
oai_set_id=oai_set_id,
func=1)
else:
return bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "OAI set does not exist.")
elif func in ["1", 1]:
res = delete_oai_set(oai_set_id)
if res[0] == 1:
out += bibharvest_templates.tmpl_print_info(CFG_SITE_LANG, "OAI set definition #%s deleted." % oai_set_id)
out += "<br />"
else:
pass
lnargs = [["ln", ln]]
out += "<br /><br />"
- out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, weburl = weburl, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs )
+ out += bibharvest_templates.tmpl_link_with_args(ln = CFG_SITE_LANG, funcurl = "admin/bibharvest/oaiarchiveadmin.py/index", title = "Return to main selection", args = lnargs )
from invenio.bibrankadminlib import write_outcome,modify_translations,get_def_name,get_i8n_name,get_name,get_rnk_nametypes,get_languages,check_user,is_adminuser,addadminbox,tupletotable,tupletotable_onlyselected,addcheckboxes,createhiddenform
from invenio.dbquery import run_sql, get_table_status_info
from invenio.webpage import page, pageheaderonly, pagefooteronly, adderrorbox
from invenio.webuser import getUid, get_email
from invenio.bibindex_engine_stemmer import get_stemming_language_map
<td>1. <small><a href="%s/admin/bibindex/bibindexadmin.py/field?ln=%s&mtype=perform_showfieldoverview#1">Overview of logical fields</a></small></td>
<td>1. <small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&ln=%s&mtype=perform_modifyfield">Modify field code</a></small></td>
<td>3. <small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&ln=%s&mtype=perform_modifyfieldtags">Modify MARC tags</a></small></td>
<td>5. <small><a href="%s/admin/bibindex/bibindexadmin.py/editfield?fldID=%s&ln=%s&mtype=perform_showdetailsfield">Show field usage</a></small></td>
<td>1. <small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&ln=%s&mtype=perform_modifyindex">Modify index name / descriptor</a></small></td>
<td>3. <small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&ln=%s&mtype=perform_modifyindexfields">Modify index fields</a></small></td>
<td>4. <small><a href="%s/admin/bibindex/bibindexadmin.py/editindex?idxID=%s&ln=%s&mtype=perform_modifyindexstemming">Modify index stemming language</a></small></td>
text += """<b><span class="info">By deleting an index, you may also loose any indexed data in the forward and reverse table for this index.</span></b><br />"""
text += """Do you want to delete the index '%s' and all its relations and definitions.""" % (idx[0][1])
query = "SELECT tag.value FROM tag, field_tag WHERE tag.id=field_tag.id_tag AND field_tag.id_field=%d ORDER BY field_tag.score DESC,tag.value ASC" % field_id
if confirm in [0, "0"] and get_idx(idxID)[0][4] == idxSTEM:
output += """<span class="info">Stemming language has not been changed</span>"""
elif confirm in [0, "0"]:
text = """
<span class="important">You are going to change the stemming language for this index. Please note you should not enable stemming for structured-data indexes like "report number", "year", "author" or "collection". On the contrary, it is advisable to enable stemming for indexes like "fulltext", "abstract", "title", etc. since this would improve retrieval quality. <br /> Beware that after changing the stemming language of an index you will have to reindex it. It is a good idea to change the stemming language and to reindex during low usage hours of your service, since searching will be not functional until the reindexing will be completed</span>.<br /> <strong>Are you sure you want to change the stemming language of this index?</strong>"""
- subtitle = """<a name="4"></a>3. Modify MARC tags for the logical field '%s' <small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[int(fldID)], weburl)
+ subtitle = """<a name="4"></a>3. Modify MARC tags for the logical field '%s' <small>[<a title="See guide" href="%s/help/admin/bibindex-admin-guide">?</a>]</small>""" % (fld_dict[int(fldID)], CFG_SITE_URL)
output = """<dl>
<dt>Menu</dt>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/addtag?fldID=%s&ln=%s#4.1">Add MARC tag</a></dd>
<dd><a href="%s/admin/bibindex/bibindexadmin.py/deletetag?fldID=%s&ln=%s#4.1">Delete unused MARC tags</a></dd>
<p>BibMatch allows to match bibliographic data in a MARCXML file against database content. Providing an input file in MARCXML the produced output corresponds to a selection of records in MARCXML that do (not) match the database content. This way it is for example possible to identify potential double entries before they get uploaded in a database. </p>
<p>The bibrank web interface enables you to modify the configuration of most aspects of BibRank. For full functionality, it is advised to
let the http-daemon have write/read access to your invenio/etc/bibrank directory. If this is not wanted, you have to edit the configuration files from the console using your favourite text editor.</p>
<a name="mi"></a><h3>3.1 Main interface</h3>
<p>In the main interface screen, you see a list of all rank methods currently added. Each rank method is identified by the rank method code. To find out about the functionality available, check out the topics below.</p>
<b>Explanation of concepts</b>
<blockquote>
<pre>
Rank method:
A method responsible for creating the necessary data to rank a result.
Translations:
Each rank method may have many names in many languages.
Collections:
Which collections the rank method should be visible in.
</pre>
</blockquote>
<a name="ar"></a><h3>3.2 Add rank method</h3>
<p>When pressing the link in the upper right corner from the main interface, you will see the interface for adding a new rank method. The two available options that needs to be decided upon, are the bibrank code and the template to use, both values can be changed later. The bibrank code is used by the bibrank daemon to run the method, and should be fairly short without spaces. Which template you are using, decides how the ranking will be done, and must before used, be changed to suit your CDS Invenio configuration. When confirming to add a new rank method, it will be added to the list of possible rank methods, and a configuration file will be created if the httpd user has proper rights to the 'invenio/etc/bibrank' directory. If not, the file has to manually be created with the name 'bibrankcode.cfg' where bibrankcode is the same as given in the interface.</p>
<a name="sd"></a><h3>3.3 Show details of rank method</h3>
<p>This interface gives you an overview of the current status of the rank method, and gives direct access to the various interfaces for changing the configuration.
In the overview section, you see the bibrank code, for use with the bibrank daemon, and the date for the last run of the rank method.
In the statistics section you see how many records have been added to the rank method and other statistic data. In the collection part, the collections which the rank method is visible to is shown. The translations part shows the various translations in the languages available in CDS Invenio. On the bottom the configuration file is shown, if accessible.</p>
<a name="mr"></a><h3>3.4 Modify rank method</h3>
<p>This interface gives access to modify the bibrank code given when creating the rank method and the configuration file of the rank method, if the file can be accessed. If not, it may not exist, or the httpd user doesn't have enough rights to read the file. On the bottom of the interface, it is possible to choose a template, see it, and copy it over the old rank method configuration if wanted. Remember that the values present in the template is an example, and must be changed where necessary. See this documentation for information about this, and the 'BibRank Internals' link below for additional information.</p>
<a name="dr"></a><h3>3.5 Delete rank method</h3>
<p>If it is necessary to delete a rank method, some precautions must be taken since the configuration of the method will be lost. When deleting a rank method, the configuration file will also be deleted ('invenio/etc/bibrank/bibrankcode.cfg' where bibrankcode is the code of the rank method) if accessible to the httpd user. If not, the file can be deleted manually from console. Any bibrank tasks scheduled to run the deleted rank method must be modified or deleted manually.</p>
<a name="mt"></a><h3>3.6 Modify translations</h3>
<p>If you want to use internalisation of the rank method names, you have to add them using the 'Modify translations' interface. Below a list of all the languages used in the CDS Invenio installation will be shown with the possibility to add the translation for each language.</p>
<p>If a rank method should be visible to the users of the CDS Invenio search interface, it must be enabled for one or several collections. A rank method can be visible in the search interface of the whole site, or just one collection. The collections in the upper list box does not show the rank method in the search interface to the user. To change this select the wanted collection and press 'Enable' to enable the rank method for this collection. The collections that the method has been activated for, is shown in the lower list box. To remove a collection, select it and press the 'Disable' button to remove it from the list of collections which the rank method is enabled for.</p>
<a name="bd"></a><h2>4. BibRank Daemon</h2>
<p>The bibrank daemon read the necessary metadata from the CDS Invenio database and combines the read metadata
in different ways to create the ranking data necessary at searchtime to fast be able to rank the results.</p>
<a name="cli1"></a><h3>4.1 Command Line Interface</h3>
<blockquote>
<pre>
Usage bibrank:
bibrank -wjif -a --id=0-30000,30001-860000 --verbose=9
bibrank -wjif -d --modified='2002-10-27 13:57:26'
bibrank -wwrd --recalculate --collection=Articles
bibrank -wwrd -a -i 234-250,293,300-500 -u admin@localhost
Ranking options:
-w, --run=r1[,r2] runs each rank method in the order given
-c, --collection=c1[,c2] select according to collection
-i, --id=low[-high] select according to doc recID
-m, --modified=from[,to] select according to modification date
-l, --lastupdate select according to last update
-a, --add add or update words for selected records
-d, --del delete words for selected records
-S, --stat show statistics for a method
-R, --recalculate recalculate weigth data, used by word frequency method
should be used if ca 1% of the document has been changed
since last time -R was used
Repairing options:
-k, --check check consistency for all records in the table(s)
check if update of ranking data is necessary
-r, --repair try to repair all records in the table(s)
Scheduling options:
-u, --user=USER user name to store task, password needed
-s, --sleeptime=SLEEP time after which to repeat tasks (no)
e.g.: 1s, 30m, 24h, 7d
-t, --time=TIME moment for the task to be active (now)
e.g.: +15s, 5m, 3h , 2002-10-27 13:57:26
General options:
-h, --help print this help and exit
-V, --version print version and exit
-v, --verbose=LEVEL verbose level (from 0 to 9, default 1)
</pre>
</blockquote>
<a name="ubd"></a><h3>4.2 Using BibRank</h3>
<h4>Step 1 - Adding the rank option to the search interface</h4>
<p>To be able to add the needed ranking data to the database, you first have to add the rank method to the database, and
add the wished code you want to use together with it. The name of the configuration file in the next section, needs to
have the same name as the code stored in the database.</p>
<h4>Step 2 - Get necessary external data (ex. jif values)</h4>
<p>Find out what is necessary of data for each method. The bibrankgkb documentation below may be of assistance.</p>
<p>The configuration files for the different rank methods has different option, so verify that you are using the correct
configuration file and rank method. A template for each rank method exists as examples, but may not work on all configurations of CDS Invenio.
For a description of each rank method and the configuration necessary, check section 6 below.</p>
<h4>Step 4 - Add the ranking method as a scheduled task</h4>
<p>When the configuration is okay, you can add the bibrank daemon to the task scheduler using the scheduling options. The daemon can then do a update of the rank method once each day or similar automatically.</p>
<b>Example</b>
<blockquote>
<pre>
$ bibrank -wjif -r
Task #53 was successfully scheduled for execution.
</pre>
</blockquote>
<p>It is adviced to run the BibRank daemon using no parameters, since the default settings then will be used.</p>
<b>Example</b>
<blockquote>
<pre>
$ bibrank
Task #2505 was successfully scheduled for execution.
</pre>
</blockquote>
<h4>Step 5 - Running bibrank manually</h4>
<p>If BibRank is scheduled without any parameters, and no records has been modified, you may get a output like shown below.</p>
<b>Example</b>
<blockquote>
<pre>
$ bibrank 2505
2004-09-07 17:51:46 --> Task #2505 started.
2004-09-07 17:51:46 -->
2004-09-07 17:51:46 --> Running rank method: Number of downloads.
2004-09-07 17:51:47 --> No new records added since last time method was run
2004-09-07 17:52:11 --> Number of lines read from knowledgebase file: 420
2004-09-07 17:52:11 --> Number of records available in rank method: 0
2004-09-07 17:52:12 -->
2004-09-07 17:52:12 --> Running rank method: Word frequency
2004-09-07 17:52:13 --> rnkWORD01F contains 256842 words from 677912 records
2004-09-07 17:52:14 --> rnkWORD01F is in consistent state
2004-09-07 17:52:14 --> Using the last update time for the rank method
2004-09-07 17:52:14 --> No new records added. rnkWORD01F is up to date
2004-09-07 17:52:14 --> rnkWORD01F contains 256842 words from 677912 records
2004-09-07 17:52:14 --> rnkWORD01F is in consistent state
2004-09-07 17:52:14 --> Task #2505 finished.
</pre>
</blockquote>
<h4>Step 6 - Fast update of modified records</h4>
<p>If you just want to update the latest additions or modified records, you may want to do a faster update by running the daemon without the recalculate option. (the recalculate option is off by default). This may cause lower accurancy when ranking.</p>
<a name="brm"></a><h2>5. BibRank Methods</h2>
<p>Each BibRank method has a configuration file which contains different parameters and sections necessary to do the ranking.</p>
<a name="str"></a><h3>5.1 Single tag rank method</h3>
<p>This method uses one MARC tag together with a file containing possible values for this MARC tag together with a ranking value. This data is used to create a structure containing the record id associated with the ranking value based on the content of the tag. The method can be used for various ways of ranking like ranking by Journal Impact Factor, or use it to let certain authors always appear top of a search.
The parameters needed to be configured for this method is the 'tag','kb_src' and 'check_mandatory_tags'.</p>
<br/><br/><b>Example</b>
<blockquote>
<pre>
[rank_method]
function = single_tag_rank_method
[single_tag_rank]
tag = 909C4p
kb_src = /home/invenio/etc/bibrank/jif.kb
check_mandatory_tags = 909C4c,909C4v,909C4y
</pre>
</blockquote>
<b>Explanation:</b>
<blockquote>
<pre>
[rank_method]
##The function which is responsible for doing the work. Should not be changed
function = single_tag_rank_method
##This section must be available if the single_tag_rank_method is going to be used
[single_tag_kb]
##The tag which got the value to be searched for on the left side in the kb file (like the journal name)
tag = 909C4p
##The path to the kb file which got the content of the tag above on left side, and value on the left side
kb_src = /home/invenio/etc/bibrank/jif.kb
##Tags that must be included for a record to be added to the ranking data, to disable remove tags
<p>The kb_src file must contain data on the form:</p>
<blockquote>
<pre>
Phys. Rev., D---3.838
Phys. Rev. Lett.---6.462
Phys. Lett., B---4.213
Nucl. Instrum. Methods Phys. Res., A---0.964
J. High Energy Phys.---8.664
</pre>
</blockquote>
<p>The left side must match the content of the tag mentioned in the tag variable.
<a name="wrd"></a><h3>5.2 Word Similarity/Similar Records</h3>
<p>The Word Similarity/Similar Records method uses the content of the tags selected to determine which records is most relevant to a query, or most similar to a selected record. This method got a lot of parameters to configure, and it may need some tweaking to get the best result. The BibRank code for this method has to be 'wrd' for it to work. For best result, it is adviced to install the stemming module mentioned in INSTALL, and use a stopword list containing stopwords in the languages the records exists in. The stemmer and stopword list is used to get better results and to limit the size of the index, thus making ranking faster and more accurate. For best result with the stemmer, it is important to mark each tag to be used with the most common language the value of the tag may be in. It is adviced to not change the 'function','table' and the parameters under [find_similar]. If the stemmer is not installed, to assure that no problems exists, the 'stem_if_avail' parameter should be set to 'no'. Each tag to be used by the method has to be given a point. The number of points describes how important one word is in this tag.</p>
<p>When running BibRank to update the index for this rank method, it is not necessary to recalculate each time, but when large number of records has been updated/added, it can be wise to recalculate using the recalculate parameter of BibRank.</p>
<b>Example</b>
<blockquote>
<pre>
[rank_method]
function = word_similarity
[word_similarity]
stemming = en
table = rnkWORD01F
stopword = True
relevance_number_output_prologue = (
relevance_number_output_epilogue = )
#MARC tag,tag points, tag language
tag1 = 6531_a, 2, en
tag2 = 695__a, 1, en
tag3 = 6532_a, 1, en
tag4 = 245__%, 10, en
tag5 = 246_%, 1, fr
tag6 = 250__a, 1, en
tag7 = 711__a, 1, en
tag8 = 210__a, 1, en
tag9 = 222__a, 1, en
tag10 = 520__%, 1, en
tag11 = 590__%, 1, fr
tag12 = 111__a, 1, en
tag13 = 100__%, 2, none
tag14 = 700__%, 1, none
tag15 = 721__a, 1, none
[find_similar]
max_word_occurence = 0.05
min_word_occurence = 0.00
min_word_length = 3
min_nr_words_docs = 3
max_nr_words_upper = 20
max_nr_words_lower = 10
default_min_relevance = 75
</pre>
</blockquote>
<b>Explanation:</b>
<blockquote>
<pre>
[rank_method]
#internal name for the bibrank program, do not modify
function = word_similarity
[word_similarity]
#if stemmer is available, default stemminglanguage should be given here. Adviced to turn off if not installed
stemming = en
#the internal table to load the index tables from.
table = rnkWORD01F
#remove stopwords?
stopword = True
#text to show before the rank value when the search result is presented. <-- to hide result
relevance_number_output_prologue = (
#text to show after the rank value when the search result is presented. --> to hide result
relevance_number_output_epilogue = )
#MARC tag,tag points, tag language
#a list of the tags to be used, together with a number describing the importance of the tag, and the
#most common language for the content. Not all languages are supported. Among the supported ones are:
tag4 = 245__%, 10, en #title, the words in the title is usually describing a record very good.
tag5 = 246_% , 1, fr #french title
tag6 = 250__a, 1, en #title
tag7 = 711__a, 1, en #title
tag8 = 210__a, 1, en #abbreviated
tag9 = 222__a, 1, en #key title
[find_similar]
#term should exist in maximum X/100% of documents
max_word_occurence = 0.05
#term should exist in minimum X/100% of documents
min_word_occurence = 0.00
#term should be atleast 3 characters long
min_word_length = 3
#term should be in atleast 3 documents or more
min_nr_words_docs = 3
#do not use more than 20 terms for "find similar"
max_nr_words_upper = 20
#if a document contains less than 10 terms, use much used terms too, if not ignore them
max_nr_words_lower = 10
#default minimum relevance value to use for find similar
default_min_relevance = 75
</pre>
</blockquote>
<p>Tip: When executing a search using a ranking method, you can add "verbose=1" to the list of parameteres
in the URL to see which terms have been used in the ranking.</p>
<a name="cmb"></a><h3>5.3 Combine method</h3>
<p>The 'Combine method' is running each method mentioned in the config file and adding the score together
based on the importance of the method given by the percentage.</p>
<b>Example</b>
<blockquote>
<pre>
[rank_method]
function = combine_method
[combine_method]
method1 = cern_jif,33
method2 = cern_acc,33
method3 = wrd,33
relevance_number_output_prologue = (
relevance_number_output_epilogue = )
</pre>
</blockquote>
<b>Explanation:</b>
<blockquote>
<pre>
[rank_method]
#tells which method to use, do not change
function = combine_method
[combine_method]
#each line tells which method to use, the code is the same as in the BibRank interface, the number describes how
#much of the total score the method should count.
method1 = jif,50
method2 = wrd,50
#text to be shown before the rank value on the search result screen.
relevance_number_output_prologue = (
#text to be shown after the rank value on the search result screen.
relevance_number_output_epilogue = )
</pre>
</blockquote>
<a name="bt"></a><h2>6. bibrankgkb Tool</h2>
<p>For some ranking methods, like the single_tag_rank method, a knowledge base file (kb) with the needed data in the correct format is necessary. This file can be created using the bibrankgkb tool which can read the data either from
the CDS Invenio database, from several web pages using regular expressions, or from another file. In case one source
has another naming convention, bibrank can convert between them using a convert file.</p>
<a name="cli2"></a><h3>6.1 Command Line Interface</h3>
-i, --input=file input file, default from /etc/bibrank/bibrankgkb.cfg
-o, --output=file output file, will be placed in current folder
General options:
-h, --help print this help and exit
-V, --version print version and exit
-v, --verbose=LEVEL verbose level (from 0 to 9, default 1)
</pre>
</blockquote>
<a name="ubt"></a><h3>6.2 Using bibrankgkb</h3>
<h4>Step 1 - Find sources</h4>
<p>Since some of the data used for ranking purposes is not freely available, it cannot be bundled with CDS Invenio. To get hold of the necessary data,
you may find it useful to ask your library if they have a copy of the data that can be used (like the Journal Impact Factors from the Science Citation Index), or use google to search the web for any public source.</p>
<h4>Step 2 - Create configuration file</h4>
<p>The default configuration file is shown below.</p>
<blockquote>
<pre>
##The main section
[bibrankgkb]
##The url to a web page with the data to be read, does not need to have the same name as this one, but if there
are several links, the url parameter should end with _0->
<p>When you have found a source for the data, created the configuration file, it may be necessary to
create an convertion file, but this depends on the conversions used in the available data versus
the convertion used in your CDS Invenio installation.</p>
<p>The available data may look like this:</p>
<blockquote>
<pre>
COLLOID SURFACE A---1.98
</pre>
</blockquote>
<p>But in CDS Invenio you are using:</p>
<blockquote>
<pre>
Colloids Surf., A---1.98
</pre>
</blockquote>
<p>By using a convertion file like:</p>
<blockquote>
<pre>
COLLOID SURFACE A---Colloids Surf., A
</pre>
</blockquote>
<p>You can convert the source to the correct naming convention.</p>
<blockquote>
<pre>
Colloids Surf., A---1.98
</pre>
</blockquote>
<h4>Step 3 - Run tool</h4>
<p>When ready to run the tool, you may either use the default file (/etc/bibrank/bibrankgkb.cfg), or use another one by giving it using the input variable '--input'.
If you want to test the configuration, you can use '--verbose=9' to output on screen, or if you want to save it to a file, use
'--output=filename', but remember that the file will be saved in the program directory.</p>
<p>This page summarizes all the information suitable to dig inside
BibRank and it's different ranking methods.</p>
<blockquote>
<dl>
<dt><a href="bibrank-single-tag-rank">Single tag rank method</a>
<dd>Ex. of use: Journal Impact Factor. Explains what this ranking method<br />
does to create the ranking data for use when ranking based on<br />
Journal Impact Factor or similar methods.</dd>
</dt>
<dt><a href="bibrank-word-similarity">Word Similarity/Similar Records methods</a>
<dd>Ex. of use: Ranking based on the importance of words in the whole<br />
collection and in a single record. Explains how the ranking data is calculated.</dd>
</dt>
<dt><a href="bibrank-api">Bibrank Record Sorter API</a>
<dd>How to call the Bibrank Record Sorter library from your python programs if necessary.</dd></dt>
<dt><a href="bibrank-bibrankgkb">Create knowledgebase files (bibrankgkb)</a> <dd>Explains how to create knowledgebase files using the bibrankgkb tool</dd></dt>
"""Return list of field values for field tag inside record RECID."""
out = []
if tag == "001___":
out.append(str(recID))
else:
digit = tag[0:2]
bx = "bib%sx" % digit
bibx = "bibrec_bib%sx" % digit
query = "SELECT bx.value FROM %s AS bx, %s AS bibx WHERE bibx.id_bibrec='%s' AND bx.id=bibx.id_bibxxx AND bx.tag LIKE '%s'" "ORDER BY bibx.field_number, bx.tag ASC" % (bx, bibx, recID, tag)
sql_query = "select count(*) from rnkDOWNLOADS where id_bibrec=%s %s and download_time>='%s' and download_time<'%s';" % (id_bibrec, id_bibdoc_query_addition, date1, date2)
res = run_sql(sql_query)[0][0]
list_tuple.append((elem[0], res))
#list_tuple = sort_list_tuple_by_date(list_tuple)
return (list_tuple)
def sort_list_tuple_by_date(list_tuple):
"""Sort a list of tuple of the forme ("09/2004", 3)according to the
<dt>The rank method is currently enabled for these collections:</dt>
<dd>
"""
col_list = get_rnk_col(rnkID, ln)
if not col_list:
output += """No collections"""
else:
for (id, name) in col_list:
output += """%s, """ % name
output += """</dd>
</dl>
"""
col_list = get_def_name('', "collection")
col_rnk = dict(get_rnk_col(rnkID))
col_list = filter(lambda x: not col_rnk.has_key(x[0]), col_list)
if col_list:
text = """
<span class="adminlabel">Enable for:</span>
<select name="colID" class="admin_w200">
<option value="">- select collection -</option>
"""
for (id, name) in col_list:
text += """<option value="%s" %s>%s</option>""" % (id, (func in ["0", 0] and confirm in ["0", 0] and colID and int(colID) == int(id)) and 'selected="selected"' or '' , name)
elif confirm in ["1", 1] and func in ["0", 0] and colID:
subtitle = "Step 3 - Result"
output += write_outcome(finresult)
elif confirm not in ["0", 0] and func in ["0", 0]:
output += """<b><span class="info">Please select a collection.</span></b>"""
col_list = get_rnk_col(rnkID, ln)
if col_list:
text = """
<span class="adminlabel">Disable for:</span>
<select name="colID" class="admin_w200">
<option value="">- select collection -</option>
"""
for (id, name) in col_list:
text += """<option value="%s" %s>%s</option>""" % (id, (func in ["1", 1] and confirm in ["0", 0] and colID and int(colID) == int(id)) and 'selected="selected"' or '' , name)
"""form to add a new rank method with these values:"""
subtitle = 'Step 1 - Create new rank method'
output = """
<dl>
<dt>BibRank code:</dt>
<dd>A unique code that identifies a rank method, is used when running the bibrank daemon and used to name the configuration file for the method.
<br />The template files includes the necessary parameters for the chosen rank method, and only needs to be edited with the correct tags and paths.
<br />For more information, please go to the <a title="See guide" href="%s/help/admin/bibrank-admin-guide">BibRank guide</a> and read the section about adding a rank method</dd>
text += """<b><span class="info"><br />Configuration file created using '%s' as template.</span></b>""" % template
else:
text += """<b><span class="info"><br />Empty configuration file created.</span></b>"""
except StandardError, e:
text += """<b><span class="info"><br />Sorry, could not create configuration file: '%s/bibrank/%s.cfg', either because it already exists, or not enough rights to create file. <br />Please create the file in the path given.</span></b>""" % (CFG_ETCDIR, get_rnk_code(rnkID)[0][0])
else:
text = """<b><span class="info">Sorry, could not add rank method, rank method with the same BibRank code probably exists.</span></b>"""
output += text
elif not rnkcode and confirm not in [-1, "-1"]:
output += """<b><span class="info">Sorry, could not add rank method, not enough data submitted.</span></b>"""
<dd>When changing the BibRank code of a rank method, you must also change any scheduled tasks using the old value.
<br />For more information, please go to the <a title="See guide" href="%s/help/admin/bibrank-admin-guide">BibRank guide</a> and read the section about modifying a rank method's BibRank code.</dd>
text += """<b><span class="info">Cannot load file, either it does not exist, or not enough rights to read it: '%s/bibrank/%s.cfg'<br />Please create the file in the path given.</span></b>""" % (CFG_ETCDIR, get_rnk_code(rnkID)[0][0])
output += createhiddenform(action="modifyrank",
text=text,
rnkID=rnkID,
button="Modify",
confirm=1)
if rnkcode and confirm in ["1", 1] and get_rnk_code(rnkID)[0][0] != rnkcode:
oldcode = get_rnk_code(rnkID)[0][0]
result = modify_rnk(rnkID, rnkcode)
subtitle = "Step 3 - Result"
if result:
text = """<b><span class="info">Rank method modified.</span></b>"""
text = """<b><span class="info">Sorry, could not change name of cfg file, must be done manually: '%s/bibrank/%s.cfg'</span></b>""" % (CFG_ETCDIR, oldcode)
else:
text = """<b><span class="info">Sorry, could not modify rank method.</span></b>"""
text = """<b><span class="info"><br />Sorry, could not modify configuration file, please check for rights to do so: '%s/bibrank/%s.cfg'<br />Please modify the file manually.</span></b>""" % (CFG_ETCDIR, get_rnk_code(rnkID)[0][0])
<dd><strong>When deleting a rank method, you also deletes all data related to the rank method, like translations, which collections
it was attached to and the data necessary to rank the searchresults. Any scheduled tasks using the deleted rank method will also stop working.
<br /><br />For more information, please go to the <a title="See guide" href="%s/help/admin/bibrank-admin-guide">BibRank guide</a> and read the section regarding deleting a rank method.</strong></dd>
text += """<br /><b><span class="info">Sorry, could not delete configuration file: '%s/bibrank/%s.cfg'.</span><br />Please delete the file manually.</span></b>""" % (CFG_ETCDIR, rnkcode)
else:
text = """<b><span class="info">Sorry, could not delete rank method</span></b>"""
except StandardError, e:
text = """<b><span class="info">Sorry, could not delete rank method, most likely already deleted</span></b>"""
This is the fulltext version of my thesis in the PDF format.
Chapter 5 still needs some revision.
</subfield>
</datafield>
</record>
</pre>
</p>
<p>The FFT tag can be repetitive, so one can pass along another FFT
tag instance containing a pointer to e.g. the thesis defence slides.
The subfields of an FFT tag are non-repetitive.
</p>
<p>The bibupload process, when it encounters FFT tags, will
automatically populate fulltext storage space
(<code>/opt/cds-invenio/var/data/files</code>) and metadata record
associated tables (<code>bibrec_bibdoc</code>, <code>bibdoc</code>) as
appropriate. It will also enrich the 856 tags (URL tags) of the MARC
metadata of the record in question with references to the latest
versions of each file.
</p>
<p>The bibupload process supports the usual modes correct, append,
replace, insert with a semantic that is somewhat similar to the
semantic of the metadata upload:
<blockquote>
<table border="1">
<thead>
<tr>
<th></th><th>Metadata</th> <th>Fulltext</th>
</tr>
</thead>
<tbody>
<tr>
<td>objects being uploaded</td><td> MARC field instances characterized by tags (010-999) </td> <td> fulltext files characterized by unique file names (FFT $n)</td>
</tr>
<tr>
<td> insert </td><td> insert new record; must not exist </td><td> insert new files; must not exist </td>
</tr>
<tr>
<td> append </td><td> append new tag instances for the given tag XXX, regardless of existing tag instances</td><td> append new files, if filename (i.e. new format) not already present </td>
</tr>
<tr>
<td> correct </td><td> correct tag instances for the given tag XXX; delete existing ones and replace with given ones </td><td> correct files with the given filename; add new revision or delete file</td>
</tr>
<tr>
<td> replace </td><td> replace all tags, whatever XXX are</td><td> replace all files, whatever filenames are </td>
</tr>
</tbody>
</table>
</blockquote>
</p>
<p>Note, in append and insert mode, <pre>$m</pre> is ignored.
<p>In order to purge previous file revisions (i.e. in order to keep
only the latest file version), please use the correct mode with $n
docname and $t PURGE as the special keyword.
</p>
<p>In order to delete all existing versions of a file, making it
effectively hidden, please use the correct mode with $n docname
and $t DELETE as the special keyword.
</p>
<p>In order to expunge (i.e. remove completely, also from the filesystem) all existing versions of a file, making it
effectively disappear, please use the correct mode with $n docname
and $t EXPUNGE as the special keyword.
</p>
<p>In order to revert to a previous file revision (i.e. to create a new revision with
the same content as some previous revision had), please use the correct mode with
$n docname, $t REVERT as the special keyword and $v the number corresponding
to the desired version.</p>
<p>In order to preserve previous comments and descriptions when correcting, please use the KEEP-OLD-VALUE special keyword with the desired $d and $z subfield.
</p>
<p>In order to add an icon representing a document, you must use a $x subfields. All the FFT for different format of the document must have then the same $x subfield. You can use KEEP-OLD-VALUE in order to keep the previous icon when correcting.
</p>
<p>The $r subfield can contain a keyword that can be use to restrict the given document. The same keyword must be specified for all the format of a given document. The keyword will be used as the status parameter for the "viewrestrdoc" action, which can be used to give access right/restriction to desired user. e.g. if you set the keyword "thesis", you can the connect the "thesisviewer" to the action "viewrestrdoc" with parameter "status" set to "thesis". Then all the user which are linked with the "thesisviewer" role will be able to download the document. Instead any other user will not be allowed. Note, if you use the keyword "KEEP-OLD-VALUE" the previous restrictions if applicable will be kept.
def _add_new_format(bibdoc, url, format, docname, doctype, newname):
"""Adds a new format for a given bibdoc. Returns True when everything's fine."""
try:
tmpurl = download_url(url, format)
try:
bibdoc.add_file_new_format(tmpurl)
except StandardError, e:
write_message("('%s', '%s', '%s', '%s', '%s') not inserted because format already exists (%s)." % (url, format, docname, doctype, newname, e), stream=sys.stderr)
os.remove(tmpurl)
return False
except Exception, e:
write_message("Error in downloading '%s' because of: %s" % (url, e))
return False
return True
def _add_new_version(bibdoc, url, format, docname, doctype, newname):
"""Adds a new version for a given bibdoc. Returns True when everything's fine."""
try:
tmpurl = download_url(url, format)
try:
bibdoc.add_file_new_version(tmpurl)
except StandardError, e:
write_message("('%s', '%s', '%s', '%s', '%s') not inserted because '%s'." % (url, format, docname, doctype, newname, e), stream=sys.stderr)
os.remove(tmpurl)
return False
except Exception, e:
write_message("Error in downloading '%s' because of: %s" % (url, e))
return False
return True
def _add_new_icon(bibdoc, url, restriction):
"""Adds a new icon to an existing bibdoc, replacing the previous one if it exists. If url is empty, just remove the current icon."""
if not url:
bibdoc.delete_icon()
else:
try:
path = urllib2.urlparse.urlsplit(url)[2]
filename = os.path.split(path)[-1]
format = filename[len(file_strip_ext(filename)):].lower()
write_message("fft '%s' specifies a different doctype from previous fft with docname '%s'" % (str(fft), name), stream=sys.stderr)
break
if newname2 != newname:
write_message("fft '%s' specifies a different newname from previous fft with docname '%s'" % (str(fft), name), stream=sys.stderr)
break
if restriction2 != restriction:
write_message("fft '%s' specifies a different restriction from previous fft with docname '%s'" % (str(fft), name), stream=sys.stderr)
break
if icon2 != icon:
write_message("fft '%x' specifies a different icon than the previous fft with docname '%s'" % (str(fft), name), stream=sys.stderr)
break
if version2 != version:
write_message("fft '%x' specifies a different version than the previous fft with docname '%s'" % (str(fft), name), stream=sys.stderr)
break
for (url2, format2) in urls:
if format == format2:
write_message("fft '%s' specifies a second file '%s' with the same format '%s' from previous fft with docname '%s'" % (str(fft), url, format, name), stream=sys.stderr)
class BibUploadReferencesModeTest(unittest.TestCase):
"""Testing references mode."""
def setUp(self):
# pylint: disable-msg=C0103
"""Initialize the MARCXML variable"""
self.test_insert = """<record>
<controlfield tag="001">123456789</controlfield>
<datafield tag ="100" ind1=" " ind2=" ">
<subfield code="a">Tester, T</subfield>
<subfield code="u">CERN</subfield>
</datafield>
</record>"""
self.test_reference = """<record>
<controlfield tag="001">123456789</controlfield>
<datafield tag =\"""" + bibupload.CFG_BIBUPLOAD_REFERENCE_TAG + """\" ind1="C" ind2="5">
<subfield code="m">M. Lüscher and P. Weisz, String excitation energies in SU(N) gauge theories beyond the free-string approximation,</subfield>
<subfield code="s">J. High Energy Phys. 07 (2004) 014</subfield>
</datafield>
</record>"""
self.test_reference_expected_xm = """<record>
<controlfield tag="001">123456789</controlfield>
<datafield tag ="100" ind1=" " ind2=" ">
<subfield code="a">Tester, T</subfield>
<subfield code="u">CERN</subfield>
</datafield>
<datafield tag =\"""" + bibupload.CFG_BIBUPLOAD_REFERENCE_TAG + """\" ind1="C" ind2="5">
<subfield code="m">M. Lüscher and P. Weisz, String excitation energies in SU(N) gauge theories beyond the free-string approximation,</subfield>
<subfield code="s">J. High Energy Phys. 07 (2004) 014</subfield>
</datafield>
</record>"""
self.test_insert_hm = """
001__ 123456789
100__ $$aTester, T$$uCERN
"""
self.test_reference_expected_hm = """
001__ 123456789
100__ $$aTester, T$$uCERN
%(reference_tag)sC5 $$mM. Lüscher and P. Weisz, String excitation energies in SU(N) gauge theories beyond the free-string approximation,$$sJ. High Energy Phys. 07 (2004) 014
res = run_sql('SELECT bd.status FROM bibrec_bibdoc as bb JOIN bibdoc as bd ON bb.id_bibdoc = bd.id WHERE bb.id_bibrec = %s AND bd.docname = %s', (recid, docname))
## CDS Invenio is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## CDS Invenio is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with CDS Invenio; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
"""ElmSubmit configuration parameters."""
__revision__ = "$Id$"
from invenio.config import CFG_SITE_ADMIN_EMAIL, \
- weburl, CFG_SITE_NAME
+ CFG_SITE_URL, CFG_SITE_NAME
# elmsubmit configuration file:
CFG_ELMSUBMIT_FILES = {
'mailprefix': 'mail',
'test_case_1': 'elmsubmit_tests_1.mbox',
'test_case_2': 'elmsubmit_tests_2.mbox',
}
# Messages we need to send to the user, before we've identified the
# correct language to talk to them in (so we assume English!):
# pylint: disable-msg=C0301
CFG_ELMSUBMIT_NOLANGMSGS = {'bad_email': 'Your email could not be parsed correctly to discover a submission. Please check your email client is functioning correctly.',
- 'bad_submission': 'The submission data that you have provided could not be parsed correctly. Please visit <%s> for a description of the correct format.' % weburl,
+ 'bad_submission': 'The submission data that you have provided could not be parsed correctly. Please visit <%s> for a description of the correct format.' % CFG_SITE_URL,
'missing_type': 'The submission data that you have provided does not contain a TYPE field. This is mandatory for all submissions.',
'unsupported_type': 'The TYPE field of your submission does not contain a recognized value.',
'missing_fields_1': 'Your submission of type',
'missing_fields_2': 'does not contain all the required fields:',
'bad_field': 'This field does not validate correctly:',
'correct_format': 'It must be formatted as follows:',
'missing_attachment': 'We could not find the following file attached to your submission email:',
'temp_problem': 'There is a temporary problem with %s\'s email submission interface. Please retry your submission again shortly.' % CFG_SITE_NAME}
to the database and provides the <code>run_sql()</code> function to
perform SQL queries and <code>run_sql_cached()</code> for rarely
changing SELECT queries. It also exports DB exceptions for the client
code to use (see below).
</p>
<h3><code>run_sql()</code> API</h3>
<p><code>run_sql()</code> signature:
<blockquote>
<pre>
def run_sql(sql, param=None, n=0, with_desc=0):
"""Run SQL on the server with PARAM and return result.
@param param: tuple of string params to insert in the query
(see notes below)
@param n: number of tuples in result (0 for unbounded)
@param with_desc: if true, will return a
DB API 7-tuple describing columns in query
@return: if SELECT, SHOW, DESCRIBE statements: tuples of data, followed
by description if parameter
provided
if INSERT: last row id.
else: SQL result as provided by database
When the site is closed for maintenance (as governed by the
config variable CFG_ACCESS_CONTROL_LEVEL_SITE), do not attempt
to run any SQL queries but return empty list immediately.
Useful to be able to have the website up while MySQL database
is down for maintenance, hot copies, table repairs, etc.
In case of problems, exceptions are returned according to the
Python DB API 2.0. The client code can import them from this
file and catch them.
"""
</pre>
</blockquote>
</p>
<p><code>run_sql()</code> normally escapes its parameters if you
pass them in a tuple. Usually the params must use the string format (<code>%s<code>):
<blockquote>
<pre>
from invenio.dbquery import run_sql
[...]
res = run_sql("SELECT id FROM collection WHERE name=%s", (c,))
if res:
colID = res[0][0]
</pre>
</blockquote>
If you want to escape the parameters yourself in the client code, you
could in principle import and make use of the
function <code>escape_string()</code>:
<blockquote>
<pre>
from invenio.dbquery import run_sql, escape_string
[...]
res = run_sql("SELECT id FROM collection WHERE name='%s'" % escape_string(c), None)
if res:
colID = res[0][0]
</pre>
</blockquote>
but beware, this function is not UTF-8 safe! So better do not use it.
</p>
<p>The <code>run_sql()</code> raises Python DB API 2.0 exceptions that
the client code should catch and handle. An example:
<blockquote>
<pre>
from invenio.dbquery import run_sql, OperationalError
[...]
query = "select citation_data from rnkCITATIONDATA"
try:
compressed_citation_dic = run_sql(query)
except OperationalError:
compressed_citation_dic = []
</pre>
</blockquote>
</p>
<p>For the list of all exceptions and the conditions when they are
raised, see <a href="http://www.python.org/dev/peps/pep-0249/">PEP 249</a>.
<h3>Note for handling date types</h3>
<p>There is an incompatibility in handling date types between MySQLdb 0.9 and MySQLdb 1.2 (while using Python 2.2 or 2.3). If a date field is in the received tuple, its format will be:</p>
<ul>
<li>string with MySQLdb 0.9</li>
<li>datetime with MySQLdb 1.2</li>
</ul>
<p>As Python 2.2 doesn't provide <code>datetime</code> class, handling of this
problem should be done for backwards compatibility reasons. The
solution is to force MySQL to convert date to a textual format:</p>
<pre>
SELECT DATE_FORMAT(date_field,'%%Y-%%m-%%d %%H:%%i:%%s') FROM table
</pre>
<p>This conversion will return a datetext format as described in <a href="miscutil-dateutils">dateutils library</a><code>(YEAR-MONTH-DAY HOUR:MINUTE:SECOND)</code>.</p>
<h3><code>run_sql_cached()</code> API</h3>
<p> If you execute a certain SELECT query often, you can
use <code>run_sql_cached()</code> that will cache its result in memory
and return it faster next time. The function signature and usage is
similar to the one known from <code>run_sql()</code>:
17: """Error(17): You have not yet confirmed the email address for the '%s' authentication method.""",
18: """Error(18): The administrator has not yet activated your account for the '%s' authentication method.""",
19: """Error(19): The site is having troubles in sending you an email for confirming your email address. The error has been logged and will be taken care of as soon as possible."""
output += """%s %s<br />""" % (system, (CFG_EXTERNAL_AUTHENTICATION[system][1] and "(Default)" or ""))
output += "<br /><b>Changing the settings:</b><br />"
output += "Currently, all changes must be done using your favourite editor, and the webserver restarted for changes to take effect. For the settings to change, either look in the guide or in access_control_config.py ."
#output += "Remember that you will be logged out as the current user."
##baskets = run_sql("SELECT basket.id, basket.name, basket.public FROM basket, user_basket WHERE id_user=%s and user_basket.id_basket=basket.id" % userID)
##output += "<table><tr>"
##for (id, name, public) in baskets:
## output += "<tr><td>%s<br />Public: %s</td></tr>" % (name, (public=="y" and "Yes" or "No"))
## basket_records = run_sql("SELECT id_record, nb_order FROM basket_record WHERE id_basket=%s" % id)
## for (id_record, nb_order) in basket_records:
## output += "<tr><td></td><td>"
## output += print_record(id_record)
## output += "</td></tr>"
##
##output += "</tr></table>"
#else:
#output += '<b><span class="info">The account id given does not exist.</span></b>'
res = run_sql("SELECT id, email FROM user WHERE id=%s", (userID, ))
output = ""
if res:
user_pref = get_user_preferences(userID)
if confirm in [1, "1"]:
if login_method:
user_pref['login_method'] = login_method
set_user_preferences(userID, user_pref)
output += "Select default login method:<br />"
text = ""
methods = CFG_EXTERNAL_AUTHENTICATION.keys()
methods.sort()
for system in methods:
text += """<input type="radio" name="login_method" value="%s" %s>%s<br />""" % (system, (user_pref['login_method'] == system and "checked" or ""), system)
</select><br />""" % ((limit_to=="all" and "selected" or ""), (limit_to=="enabled" and "selected" or ""), (limit_to=="disabled" and "selected" or ""))
text += """<span class="adminlabel">Accounts per page:</span>
<select name="maxpage" class="admin_wvar">
<option value="25" %s>25</option>
<option value="50" %s>50</option>
<option value="100" %s>100</option>
<option value="250" %s>250</option>
<option value="500" %s>500</option>
<option value="1000" %s>1000</option>
</select><br />""" % ((maxpage==25 and "selected" or ""), (maxpage==50 and "selected" or ""), (maxpage==100 and "selected" or ""), (maxpage==250 and "selected" or ""), (maxpage==500 and "selected" or ""), (maxpage==1000 and "selected" or ""))
return run_sql('update user_query_basket set date_lastrun=%s where id_user=%s and id_query=%s and id_basket=%s;', (strftime("%Y-%m-%d"), alert[0], alert[1], alert[2],))
def get_alert_queries(frequency):
return run_sql('select distinct id, urlargs from query q, user_query_basket uqb where q.id=uqb.id_query and uqb.frequency=%s and uqb.date_lastrun <= now();', (frequency,))
def get_alert_queries_for_user(uid):
return run_sql('select distinct id, urlargs, uqb.frequency from query q, user_query_basket uqb where q.id=uqb.id_query and uqb.id_user=%s and uqb.date_lastrun <= now();', (uid,))
def get_alerts(query, frequency):
r = run_sql('select id_user, id_query, id_basket, frequency, date_lastrun, alert_name, notification from user_query_basket where id_query=%s and frequency=%s;', (query['id_query'], frequency,))
- 'ln' *string* - The language to display the interface in
- 'alerts' *array* - The existing alerts:
- 'queryid' *string* - The id of the associated query
- 'queryargs' *string* - The query string
- 'textargs' *string* - The textual description of the query string
- 'userid' *string* - The user id
- 'basketid' *string* - The basket id
- 'basketname' *string* - The basket name
- 'alertname' *string* - The alert name
- 'frequency' *string* - The frequency of alert running ('day', 'week', 'month')
- 'notification' *string* - If notification should be sent by email ('y', 'n')
- 'created' *string* - The date of alert creation
- 'lastrun' *string* - The last running date
- 'guest' *bool* - If the user is a guest user
- 'guesttxt' *string* - The HTML content of the warning box for guest users (produced by webaccount.tmpl_warning_guest_user)
"""
# load the right message language
_ = gettext_set_language(ln)
out = '<p>' + _("Set a new alert from %(x_url1_open)syour searches%(x_url1_close)s, the %(x_url2_open)spopular searches%(x_url2_close)s, or the input form.") + '</p>'
out %= {'x_url1_open': '<a href="display?ln=' + ln + '">',
out = _("You have %(x_nb_perso)s personal baskets and are subscribed to %(x_nb_group)s group baskets and %(x_nb_public)s other users public baskets.") %\
subject = "An error report has been sent from a user"
body = '''
The following comment has been reported a total of %(cmt_reported)s times.
Author: nickname = %(nickname)s
email = %(user_email)s
user_id = %(uid)s
This user has:
total number of reports = %(user_nb_abuse_reports)s
%(votes)s
Comment: comment_id = %(cmt_id)s
record_id = %(id_bibrec)s
date written = %(cmt_date)s
nb reports = %(cmt_reported)s
%(review_stuff)s
body =
---start body---
%(cmt_body)s
---end body---
Please go to the WebComment Admin interface %(comment_admin_link)s to delete this message if necessary. A warning will be sent to the user in question.''' % \
note_label = _("Note: Your nickname, %s, will be displayed as the author of this review.")
note_label %= ('<i>' + nickname + '</i>')
else:
(uid, nickname, display) = get_user_info(uid)
link = '<a href="%s/youraccount/edit">' % CFG_SITE_SECURE_URL
note_label = _("Note: you have not %(x_url_open)sdefined your nickname%(x_url_close)s. %(x_nickname)s will be displayed as the author of this comment.") % \
Please consult the section below entitled "Recalculate ranking
weights" for more details.
</p>
<p>Note that the BibSched daemon automatic mode stops as soon as some
of the tasks ends with an error. It it therefore a good idea to
inspect BibSched queue from time to time, say several times per day.
This can be done by running the BibSched command-line monitor:
<blockquote>
<pre>
$ bibsched
</pre>
</blockquote>
</p>
<p>The monitor will permit you to stop/start the daemon mode, to
delete the tasks already submitted, to run some of the tasks manually,
etc. Note also that BibSched daemon writes log and error files on its
operation and on the operation of its tasks. The log and error files
can be found at <code>/opt/cds-invenio/var/log</code>.
</p>
<p>You may want to clean up the old BibSched tasks with the DONE
status, let us say once per month, to make the task table slim and the
bibsched daemon both faster and less memory hungry:
<blockquote>
<pre>
$ echo "DELETE FROM schTASK WHERE status='DONE' AND runtime<DATE_SUB(NOW(), INTERVAL 7 DAY);" | dbexec
</pre>
</blockquote>
</p>
<h2>Recalculate ranking weights</h2>
<p>When you are adding new records to the system, the word frequency
ranking weights for old records aren't recalculated by default in
order to speed up the insertion of new records. This may influence a
bit the precision of word similarity searches. It is therefore
advised to expressely run bibrank in the recalculating mode once in a
while by doing:
<blockquote>
<pre>
$ bibrank -R -wwrd
</pre>
</blockquote>
You may want to do this either (i) periodically, say once per month
(see the previous section), or (ii) depending on the frequency of new
additions to the record database, say when the size grows by 2-3
percent.
</p>
<h2>Guest Users Cleanup</h2>
<p>Guest users create a lot of entries in CDS Invenio tables that are
related to their web sessions, their search history, personal baskets,
etc. This data has to be garbage-collected periodically. You can run this,
say every day, via:
<blockquote>
<pre>
$ inveniogc -s1d
</pre>
</blockquote>
</p>
<h2>Cleaning Up the Filesystem</h2>
<p>Different temporary log and err files are created in <code>/opt/cds-invenio/var/log</code>
and <code>/opt/cds-invenio/var/tmp</code> directory that is good to clean up from time to time. The previous command could be used to clean those files, too, via:
<blockquote>
<pre>
$ inveniogc -s1d -d
</pre>
</blockquote>
</p>
<h2>Alert Engine</h2>
<p>CDS Invenio users may set up an automatic notification email alerts
that would send them documents corresponding to the user profile by
email either daily, weekly, or monthly. It is the job of the alert
engine to do this. The alert engine has to be run every day:
<blockquote>
<pre>
$ alertengine
</pre>
</blockquote>
</p>
<p>You may want to set up an external cron job to call
A brief description of things we strive at, more or less unsuccessfully.
1. Packaging
We use the classical GNU Autoconf/Automake approach, for tutorial
see e.g. <a href="http://www.amath.washington.edu/~lf/tutorials/autoconf/tutorial_toc.html">Learning the GNU development tools</a> or the <a href="http://sources.redhat.com/autobook/autobook/autobook_toc.html">AutoBook</a>.
2. Modules
CDS Invenio started as a set of pretty independent modules developed by
independent people with independent styles. This was even more
pronounced by the original use of many different languages
(e.g. Python, PHP, Perl). Now the CDS Invenio code base is striving to
use Python everywhere, except in speed-critical parts when a
compiled language such as Common Lisp may come to the rescue in the
near future.
When modifying an existing module, we propose to strictly continue
using whatever coding style the module was originally written into.
When writing new modules, we propose to stick to the
below-mentioned standards.
The code integration across modules is happening, but is slow.
Therefore, don't be surprised to see that there is a lot of room to
refactor.
3. WML/ePerl/etc
This is not so important, because not many lines-of-code were
written in WML/ePerl. We prefer to loosely follow the GNU way, as
always.
4. Python
We aim at following recommendations from <a
href="http://www.python.org/peps/pep-0008.html">PEP 8</a>, although
the existing code surely do not fulfil them here and there.
The code indentation is done via spaces only, please do not use
tabs. One tab counts as four spaces. Emacs users can look into
- our <a href="<WEBURL>/hacking/cdsware.el">cdsware.el</a> for inspiration.
+ our <a href="<CFG_SITE_URL>/hacking/cdsware.el">cdsware.el</a> for inspiration.
All the Python code should be extensively documented via
docstrings, so you can always run pydoc file.py to peruse the
file's documentation in one simple go.
Do not forget to run pylint on your code to check for errors like
uninitialized variables and to improve its quality and conformance
to the coding standard. If you develop in Emacs, run M-x pylint
RET on your buffers frequently. Read and implement pylint
suggestions. (Note that using lambda and friends may lead to false
pylint warnings. You can switch them off by putting block comments
of the form ``# pylint: disable-msg=C0301''.)
Do not forget to run pychecker on your code either. It is another
source code checker that catches some situations better and some
situations worse than pylint. If you develop in Emacs, run C-c C-w
(M-x py-pychecker-run RET) on your buffers frequently. (Note that
using psyco on classes may lead to false pychecker warnings.)
You can check the kwalitee of your code by running ``python
modules/miscutil/lib/kwalitee.py *.py'' on your files. You can
also check the code kwalitee across all the modules by running
``make kwalitee-check'' in the main source directory.
Do not hardcode magic constants in your code. Every magic string or
a number should be put into accompanying file_config.py with
symbol name beginning by cfg_modulename_*.
Clearly separate interfaces from implementation. Document your
interfaces. Do not expose to other modules anything that does not
have to be exposed. Apply principle of least information.
Create as few new library files as possible. Do not create many
nested files in nested modules; rather put all the lib files in one
dir with bibindex_foo and bibindex_bar names.
Use imperative/functional paradigm rather then OO. If you do use
OO, then stick to as simple class hierarchy as possible. Recall
that method calls and exception handling in Python are quite
expensive.
Use rather the good old foo_bar naming convention for symbols (both
variables and function names) instead of fooBar CaMelCaSe
convention. (Except for Class names where UppercaseSymbolNames are
to be used.)
Pay special attention to name your symbols descriptively. Your
code is going to be read and work with by others and its symbols
should be self-understandable without any comments and without
studying other parts of the code. For example, use proper English
words, not abbreviations that can be misspelled in many a way; use
words that go in pair (e.g. create/destroy, start/stop; never
create/stop); use self-understandable symbol names
(e.g. list_of_file_extensions rather than list2); never misname
symbols (e.g. score_list should hold the list of scores and nothing
else - if in the course of development you change the semantics of
what the symbol holds then change the symbol name too). Do not be
afraid to use long descriptive names; good editors such as Emacs
can tab-complete symbols for you.
When hacking module A, pay close attention to ressemble existing
coding convention in A, even if it is legacy-weird and even if we
use a different technique elsewhere. (Unless the whole module A is
going to be refactored, of course.)
Speed-critical parts should be profiled with pyprof. Do not forget
to use tricks like psyco.
The code should be well tested before committed. Testing is an
integral part of the development process. Test along as you
program. The testing process should be automatized via our unit
test and regression test suite infrastructures. Please read the
<a href="test-suite">test suite strategy</a> to know more.
# raise InvenioWebJournalNoArticleTemplateError(journal_name) # todo: new exception
# except InvenioWebJournalNoArticleTemplateError:
# register_exception(req=req)
# return webjournal_error_box(req,
# "Search Page Template not found",
# "Problem with the configuration for this journal.",
# "The system couldn't find the template for the search result page of this journal. This is a mandatory file and thus indicates that the journal was setup wrong or produced an internal error. If you are neither admin nor developer there is nothing you can do at this point, but send an email request. We apologize for the inconvenience.")
# title_msg="Problem with the configuration of this journal",
# msg="The system couldn't find a definition for an issue numbering system. Issue numbers conrol the date of the publication you are seing. This indicates that there is an error in the setup of this journal or the Software itself. There is nothing you can do at the moment. If you wish you can send an inquiry to the responsible developers. We apologize for the inconvenience.")
# rule_list = config_strings["rule"]
# try:
# if len(rule_list) == 0:
# raise InvenioWebJournalNoArticleRuleError()
# except InvenioWebJournalNoArticleRuleError, e:
# register_exception(req=req)
# return webjournal_error_box(req,
# "No searchable Articles",
# "Problem with the configuration of this journal",
# "The system couldn't find the definitions for different article kinds (e.g. News, Sports, etc.). If there is nothing defined, nothing can be shown and it thus indicates that there is either a problem with the setup of this journal or in the Software itself. There is nothing you can do at this moment. If you wish you can send an inquiry to the responsible developers. We apologize for the inconvenience.")
# category_rules = []
# if argd['category'] == []:
# # append all categories
# for rule_string in rule_list:
# marc = {}
# marc["category"] = rule_string.split(",")[0]
# rule = rule_string.split(",")[1]
# marc_datafield = rule.split(":")[0]
# marc["rule_match"] = rule.split(":")[1]
# marc["marc_tag"] = marc_datafield[1:4]
# marc["marc_ind1"] = (marc_datafield[4] == "_") and " " or marc_datafield[4]
# marc["marc_ind2"] = (marc_datafield[5] == "_") and " " or marc_datafield[5]
<a name="3.11"></a><h3>3.11 Configuration of related external collections</h3>
<p>You can customize each collection to provide your users an
additional source of information external to your repository: in a
<i>book</i> collection you might want for example to provide a link to
<i>Amazon</i> items corresponding to the user's query. Futhermore, for
some external services only, you can set the collection to display the
results directly in CDS Invenio search results page.
<p>The following settings are available:
<dl>
<dt>Disabled</dt>
<dd>The external collection is not shown to the user.<dd>
<dt>See also</dt>
<dd>A link to the external collection listing the items corresponding to user's query is displayed (only once a query has been performed).</dd>
<dt>External search</dt>
<dd>User can ask to perform a search in parallel on your repository and on the external collection. Results are shown in the CDS Invenio search results page. Not available for all external collections.<dd>
<dt>External search checked</dt>
<dd>Same as above, but the external collection is searched by default. Not available for all external collections.</dd>
<dl>
<p>You can also apply the settings to sub-collections, by checking the
"<i>Apply also to daughter collections</i>" checkboxes when you apply
your modifications.
<a name="3.12"></a><h3>3.12 Detailed record page options</h3>
<p>These settings let you define how the detailed view (such as <a
-href="<WEBURL>/record/1"><WEBURL>/record/1</a>) of records in this
+href="<CFG_SITE_URL>/record/1"><CFG_SITE_URL>/record/1</a>) of records in this
<br/> <a href="#howto">Foire aux questions</a>
<br/> <a href="#howto-choose-terms">Comment sélectionner vos termes de recherche de manière intelligente (en termes de vitesse)</a>
<br/> <a href="#howto-search-for-author">Comment rechercher les publications d'un auteur donné</a>
<br/> <a href="#howto-sort-pattern">Comment trier d'après un certain critère</a>
<br/> <a href="#howto-hepdoc">Comment obtenir les documents d'autres serveurs (Google, SPIRES, KEK)</a>
<br/> <a href="#howto-fulltext">Comment rechercher le contenu des fichiers</a>
<br/> <a href="#howto-citations">Comment rechercher les citations</a>
Notez que la recherche dans "tous les champs" ne s'effectue que sur les métadonnées, et non pas sur les citations, ni le contenu des documents. Il vous faut explicitement indique une recherche dans les champs <code>fulltext</code>
ou <code>reference</code> pour chercher dans ces données. Par exemple, pour trouver le terme <em>Higgs</em> dans les métadonnées, les références et le contenu des fichiers, saisissez:
"""Splits search pattern and search field into a list of independently searchable units.
- A search unit consists of '(operator, pattern, field, type, hitset)' tuples where
'operator' is set union (|), set intersection (+) or set exclusion (-);
'pattern' is either a word (e.g. muon*) or a phrase (e.g. 'nuclear physics');
'field' is either a code like 'title' or MARC tag like '100__a';
'type' is the search type ('w' for word file search, 'a' for access file search).
- Optionally, the function accepts the match type argument 'm'.
If it is set (e.g. from advanced search interface), then it
performs this kind of matching. If it is not set, then a guess is made.
'm' can have values: 'a'='all of the words', 'o'='any of the words',
'p'='phrase/substring', 'r'='regular expression',
'e'='exact value'.
- Warnings are printed on req (when not None) in case of HTML output formats."""
opfts = [] # will hold (o,p,f,t,h) units
## check arguments: if matching type phrase/string/regexp, do we have field defined?
if (m=='p' or m=='r' or m=='e') and not f:
m = 'a'
if of.startswith("h"):
print_warning(req, "This matching type cannot be used within <em>any field</em>. I will perform a word search instead." )
print_warning(req, "If you want to phrase/substring/regexp search in a specific field, e.g. inside title, then please choose <em>within title</em> search option.")
## is desired matching type set?
if m:
## A - matching type is known; good!
if m == 'e':
# A1 - exact value:
opfts.append(['+', p, f, 'a']) # '+' since we have only one unit
elif m == 'p':
# A2 - phrase/substring:
opfts.append(['+', "%" + p + "%", f, 'a']) # '+' since we have only one unit
elif m == 'r':
# A3 - regular expression:
opfts.append(['+', p, f, 'r']) # '+' since we have only one unit
elif m == 'a' or m == 'w':
# A4 - all of the words:
p = strip_accents(p) # strip accents for 'w' mode, FIXME: delete when not needed
for word in get_words_from_pattern(p):
opfts.append(['+', word, f, 'w']) # '+' in all units
elif m == 'o':
# A5 - any of the words:
p = strip_accents(p) # strip accents for 'w' mode, FIXME: delete when not needed
for word in get_words_from_pattern(p):
if len(opfts)==0:
opfts.append(['+', word, f, 'w']) # '+' in the first unit
else:
opfts.append(['|', word, f, 'w']) # '|' in further units
else:
if of.startswith("h"):
print_warning(req, "Matching type '%s' is not implemented yet." % m, "Warning")
opfts.append(['+', "%" + p + "%", f, 'a'])
else:
## B - matching type is not known: let us try to determine it by some heuristics
if f and p[0] == '"' and p[-1] == '"':
## B0 - does 'p' start and end by double quote, and is 'f' defined? => doing ACC search
opfts.append(['+', p[1:-1], f, 'a'])
elif f and p[0] == "'" and p[-1] == "'":
## B0bis - does 'p' start and end by single quote, and is 'f' defined? => doing ACC search
opfts.append(['+', '%' + p[1:-1] + '%', f, 'a'])
elif f and p[0] == "/" and p[-1] == "/":
## B0ter - does 'p' start and end by a slash, and is 'f' defined? => doing regexp search
opfts.append(['+', p[1:-1], f, 'r'])
elif f and string.find(p, ',') >= 0:
## B1 - does 'p' contain comma, and is 'f' defined? => doing ACC search
opfts.append(['+', p, f, 'a'])
elif f and str(f[0:2]).isdigit():
## B2 - does 'f' exist and starts by two digits? => doing ACC search
opfts.append(['+', p, f, 'a'])
else:
## B3 - doing WRD search, but maybe ACC too
# search units are separated by spaces unless the space is within single or double quotes
# so, let us replace temporarily any space within quotes by '__SPACE__'
p = re_pattern_single_quotes.sub(lambda x: "'"+string.replace(x.group(1), ' ', '__SPACE__')+"'", p)
p = re_pattern_double_quotes.sub(lambda x: "\""+string.replace(x.group(1), ' ', '__SPACE__')+"\"", p)
p = re_pattern_regexp_quotes.sub(lambda x: "/"+string.replace(x.group(1), ' ', '__SPACE__')+"/", p)
# wash argument:
p = re_equal.sub(":", p)
p = re_logical_and.sub(" ", p)
p = re_logical_or.sub(" |", p)
p = re_logical_not.sub(" -", p)
p = re_operators.sub(r' \1', p)
for pi in string.split(p): # iterate through separated units (or items, as "pi" stands for "p item")
pi = re_pattern_space.sub(" ", pi) # replace back '__SPACE__' by ' '
# firstly, determine set operator
if pi[0] == '+' or pi[0] == '-' or pi[0] == '|':
oi = pi[0]
pi = pi[1:]
else:
# okay, there is no operator, so let us decide what to do by default
oi = '+' # by default we are doing set intersection...
# secondly, determine search pattern and field:
if string.find(pi, ":") > 0:
fi, pi = string.split(pi, ":", 1)
else:
fi, pi = f, pi
# look also for old ALEPH field names:
if fi and CFG_WEBSEARCH_FIELDS_CONVERT.has_key(string.lower(fi)):
fi = CFG_WEBSEARCH_FIELDS_CONVERT[string.lower(fi)]
# wash 'pi' argument:
if re_quotes.match(pi):
# B3a - quotes are found => do ACC search (phrase search)
if fi:
if pi[0] == '"' and pi[-1] == '"':
pi = string.replace(pi, '"', '') # remove quote signs
opfts.append([oi, pi, fi, 'a'])
elif pi[0] == "'" and pi[-1] == "'":
pi = string.replace(pi, "'", "") # remove quote signs
opfts.append([oi, "%" + pi + "%", fi, 'a'])
else: # unbalanced quotes, so do WRD query:
opfts.append([oi, pi, fi, 'w'])
else:
# fi is not defined, look at where we are doing exact or subphrase search (single/double quotes):
if pi[0] == '"' and pi[-1] == '"':
opfts.append([oi, pi[1:-1], "anyfield", 'a'])
if of.startswith("h"):
print_warning(req, "Searching for an exact match inside any field may be slow. You may want to search for words instead, or choose to search within specific field.")
else:
# nope, subphrase in global index is not possible => change back to WRD search
pi = strip_accents(pi) # strip accents for 'w' mode, FIXME: delete when not needed
for pii in get_words_from_pattern(pi):
# since there may be '-' and other chars that we do not index in WRD
opfts.append([oi, pii, fi, 'w'])
if of.startswith("h"):
print_warning(req, "The partial phrase search does not work in any field. I'll do a boolean AND searching instead.")
print_warning(req, "If you want to do a partial phrase search in a specific field, e.g. inside title, then please choose 'within title' search option.", "Tip")
print_warning(req, "If you want to do exact phrase matching, then please use double quotes.", "Tip")
elif fi and str(fi[0]).isdigit() and str(fi[0]).isdigit():
# B3b - fi exists and starts by two digits => do ACC search
opfts.append([oi, pi, fi, 'a'])
elif fi and not get_index_id_from_field(fi):
# B3c - fi exists but there is no words table for fi => try ACC search
opfts.append([oi, pi, fi, 'a'])
elif fi and pi.startswith('/') and pi.endswith('/'):
# B3d - fi exists and slashes found => try regexp search
opfts.append([oi, pi[1:-1], fi, 'r'])
else:
# B3e - general case => do WRD search
pi = strip_accents(pi) # strip accents for 'w' mode, FIXME: delete when not needed
# then let us check the list of non-restricted "real" sons of 'cc' and compare it to 'coll':
res = run_sql("""SELECT c.name FROM collection AS c,
collection_collection AS cc,
collection AS ccc
WHERE c.id=cc.id_son AND cc.id_dad=ccc.id
AND ccc.name=%s AND cc.type='r'
AND c.restricted IS NULL""", (cc,))
l_cc_nonrestricted_sons = []
l_c = colls
for row in res:
l_cc_nonrestricted_sons.append(row[0])
l_c.sort()
l_cc_nonrestricted_sons.sort()
if l_cc_nonrestricted_sons == l_c:
colls_out_for_display = [cc] # yep, washing permitted, it is sufficient to display 'cc'
else:
colls_out_for_display = colls # nope, we need to display all 'colls' successively
# remove duplicates:
colls_out_for_display_nondups=filter(lambda x, colls_out_for_display=colls_out_for_display: colls_out_for_display[x-1] not in colls_out_for_display[x:], range(1, len(colls_out_for_display)+1))
print_warning(req, _("No match found in collection %(x_collection)s. Other public collections gave %(x_url_open)s%(x_nb_hits)d hits%(x_url_close)s.") %\
{'x_collection': '<em>' + string.join([get_coll_i18nname(coll, ln) for coll in colls], ', ') + '</em>',
"""Sort records in 'recIDs' list according sort field 'sort_field' in order 'sort_order'.
If more than one instance of 'sort_field' is found for a given record, try to choose that that is given by
'sort pattern', for example "sort by report number that starts by CERN-PS".
Note that 'sort_field' can be field code like 'author' or MARC tag like '100__a' directly."""
_ = gettext_set_language(ln)
## check arguments:
if not sort_field:
return recIDs
if len(recIDs) > CFG_WEBSEARCH_NB_RECORDS_TO_SORT:
if of.startswith('h'):
print_warning(req, _("Sorry, sorting is allowed on sets of up to %d records only. Using default sort order.") % CFG_WEBSEARCH_NB_RECORDS_TO_SORT, "Warning")
return recIDs
sort_fields = string.split(sort_field, ",")
recIDs_dict = {}
recIDs_out = []
## first deduce sorting MARC tag out of the 'sort_field' argument:
tags = []
for sort_field in sort_fields:
if sort_field and str(sort_field[0:2]).isdigit():
# sort_field starts by two digits, so this is probably a MARC tag already
tags.append(sort_field)
else:
# let us check the 'field' table
query = """SELECT DISTINCT(t.value) FROM tag AS t, field_tag AS ft, field AS f
WHERE f.code='%s' AND ft.id_field=f.id AND t.id=ft.id_tag
ORDER BY ft.score DESC""" % sort_field
res = run_sql(query)
if res:
for row in res:
tags.append(row[0])
else:
if of.startswith('h'):
print_warning(req, _("Sorry, %s does not seem to be a valid sort option. Choosing title sort instead.") % sort_field, "Error")
tags.append("245__a")
if verbose >= 3:
print_warning(req, "Sorting by tags %s." % tags)
if sort_pattern:
print_warning(req, "Sorting preferentially by %s." % sort_pattern)
## check if we have sorting tag defined:
if tags:
# fetch the necessary field values:
for recID in recIDs:
val = "" # will hold value for recID according to which sort
vals = [] # will hold all values found in sorting tag for recID
for tag in tags:
vals.extend(get_fieldvalues(recID, tag))
if sort_pattern:
# try to pick that tag value that corresponds to sort pattern
bingo = 0
for v in vals:
if v.lower().startswith(sort_pattern.lower()): # bingo!
bingo = 1
val = v
break
if not bingo: # sort_pattern not present, so add other vals after spaces
val = sort_pattern + " " + string.join(vals)
else:
# no sort pattern defined, so join them all together
val = string.join(vals)
val = strip_accents(val.lower()) # sort values regardless of accents and case
expected_text='Search term <em><SCRIPT>alert("XSS");</SCRIPT></em> inside index <em><SCRIPT>alert("XSS");</SCRIPT></em> did not match any record.'))
def test_xss_in_advanced_search(self):
"""websearch - no XSS vulnerability in advanced search"""
expected_text='Search term <em><SCRIPT>alert("XSS");</SCRIPT></em> inside index <em><SCRIPT>alert("XSS");</SCRIPT></em> did not match any record.'))
text += """<option value="%s" %s>%s</option>""" % (id, (func in ["0", 0] and confirm in ["0", 0] and int(rnkID) == int(id)) and 'selected="selected"' or '' , name)
if confirm in ["1", 1] and func in ["0", 0] and int(rnkID) != -1:
output += write_outcome(finresult)
elif confirm not in ["0", 0] and func in ["0", 0]:
output += """<b><span class="info">Please select a rank method.</span></b>"""
coll_list = get_col_rnk(colID, ln)
if coll_list:
text = """
<span class="adminlabel">Disable:</span>
<select name="rnkID" class="admin_w200">
<option value="-1">- select rank method-</option>
"""
for (id, name) in coll_list:
text += """<option value="%s" %s>%s</option>""" % (id, (func in ["1", 1] and confirm in ["0", 0] and int(rnkID) == int(id)) and 'selected="selected"' or '' , name)
if confirm not in [-1, "-1"] and not (add_son and add_dad and rtype):
output2 += """<b><span class="info">All fields must be filled.</span></b><br /><br />
"""
elif add_son and add_dad and rtype:
add_son = int(add_son)
add_dad = int(add_dad)
if confirm not in [-1, "-1"]:
if add_son == add_dad:
output2 += """<b><span class="info">Cannot add a collection as a pointer to itself.</span></b><br /><br />
"""
elif check_col(add_dad, add_son):
res = add_col_dad_son(add_dad, add_son, rtype)
output2 += write_outcome(res)
if res[0] == 1:
output2 += """<b><span class="info"><br /> The collection will appear on your website after the next webcoll run. You can either run it manually or wait until bibsched does it for you.</span></b><br /><br />
"""
else:
output2 += """<b><span class="info">Cannot add the collection '%s' as a %s subcollection of '%s' since it will either create a loop, or the association already exists.</span></b><br /><br />
""" % (col_dict[add_son], (rtype=="r" and 'regular' or 'virtual'), col_dict[add_dad])
res = run_sql("SELECT dbquery FROM collection WHERE id=%s" % colID)
dbquery = res[0][0]
if not dbquery:
dbquery = ''
reg_sons = len(get_col_tree(colID, 'r'))
vir_sons = len(get_col_tree(colID, 'v'))
if reg_sons > 1:
if dbquery:
output += "Warning: This collection got subcollections, and should because of this not have a collection query, for further explanation, check the WebSearch Guide<br />"
elif reg_sons <= 1:
if not dbquery:
output += "Warning: This collection does not have any subcollections, and should because of this have a collection query, for further explanation, check the WebSearch Guide<br />"
if switch and switch_col_treescore(tree[move_up], tree[switch]):
output += """<b><span class="info">Moved the %s collection '%s' up and '%s' down.</span></b><br /><br />
""" % ((rtype=="r" and 'regular' or 'virtual'), col_dict[tree[move_up][0]], col_dict[tree[switch][0]])
else:
output += """<b><span class="info">Could not move the %s collection '%s' up and '%s' down.</span></b><br /><br />
""" % ((rtype=="r" and 'regular' or 'virtual'), col_dict[tree[move_up][0]], col_dict[tree[switch][0]])
elif move_down:
move_down = int(move_down)
switch = find_next(tree, move_down)
if switch and switch_col_treescore(tree[move_down], tree[switch]):
output += """<b><span class="info">Moved the %s collection '%s' down and '%s' up.</span></b><br /><br />
""" % ((rtype=="r" and 'regular' or 'virtual'), col_dict[tree[move_down][0]], col_dict[tree[switch][0]])
else:
output += """<b><span class="info">Could not move the %s collection '%s' up and '%s' down.</span></b><br /><br />
""" % ((rtype=="r" and 'regular' or 'virtual'), col_dict[tree[move_up][0]],col_dict[tree[switch][0]])
elif delete:
delete = int(delete)
if confirm in [0, "0"]:
if col_dict[tree[delete][0]] != col_dict[tree[delete][3]]:
text = """<b>Do you want to remove the %s collection '%s' and its subcollections in the %s collection '%s'.</b>
""" % ((tree[delete][4]=="r" and 'regular' or 'virtual'), col_dict[tree[delete][0]], (rtype=="r" and 'regular' or 'virtual'), col_dict[tree[delete][3]])
else:
text = """<b>Do you want to remove all subcollections of the %s collection '%s'.</b>
""" % ((rtype=="r" and 'regular' or 'virtual'), col_dict[tree[delete][3]])
if move_from_id == move_to_id and move_from_rtype == move_to_rtype:
output += """<b><span class="info">Cannot move to itself.</span></b><br /><br />
"""
elif tree_from[move_from_id][3] == tree_to[move_to_id][0] and move_from_rtype==move_to_rtype:
output += """<b><span class="info">The collection is already there.</span></b><br /><br />
"""
elif check_col(tree_to[move_to_id][0], tree_from[move_from_id][0]) or (tree_to[move_to_id][0] == 1 and tree_from[move_from_id][3] == tree_to[move_to_id][0] and move_from_rtype != move_to_rtype):
text = """<b>Move %s collection '%s' to the %s collection '%s'.</b>
""" % ((tree_from[move_from_id][4]=="r" and 'regular' or 'virtual'), col_dict[tree_from[move_from_id][0]], (tree_to[move_to_id][4]=="r" and 'regular' or 'virtual'), col_dict[tree_to[move_to_id][0]])
output += """<b><span class="info">Cannot move the collection '%s' and set it as a subcollection of '%s' since it will create a loop.</span></b><br /><br />
if (move_to_id != 0 and move_col_tree(tree_from[move_from_id], tree_to[move_to_id])) or (move_to_id == 0 and move_col_tree(tree_from[move_from_id], tree_to[move_to_id], move_to_rtype)):
output += """<b><span class="info">Moved %s collection '%s' to the %s collection '%s'.</span></b><br /><br />
""" % ((move_from_rtype=="r" and 'regular' or 'virtual'), col_dict[tree_from[move_from_id][0]], (move_to_rtype=="r" and 'regular' or 'virtual'), col_dict[tree_to[move_to_id][0]])
else:
output += """<b><span class="info">Could not move %s collection '%s' to the %s collection '%s'.</span></b><br /><br />
""" % ((move_from_rtype=="r" and 'regular' or 'virtual'), col_dict[tree_from[move_from_id][0]], (move_to_rtype=="r" and 'regular' or 'virtual'), col_dict[tree_to[move_to_id][0]])
text = """Do you want to remove the %s '%s' %s from the collection '%s'.""" % (field, fld_dict[fldID], (fldvID not in["", "None"] and "with value '%s'" % fldv_dict[fldvID] or ''), col_dict[colID])
run_sql("UPDATE collection_field_fieldvalue SET score_fieldvalue=%s WHERE id_collection=%s and id_field=%s and id_fieldvalue=%s" % (vscore, colID, fldID, fldvID))
vscore -= 1
output += write_outcome((1, ""))
else:
output += write_outcome((0, (0, "No values to order")))
if CFG_SITE_NAME != run_sql("SELECT name from collection WHERE id=1")[0][0]:
res = run_sql("update collection set name='%s' where id=1" % CFG_SITE_NAME)
if res:
fin_output += """<b><span class="info">The name of the root collection has been modified to be the same as the %(sitename)s installation name given prior to installing %(sitename)s.</span><b><br />""" % {'sitename' : CFG_SITE_NAME}
<td>1. <small><a href="%s/admin/websearch/websearchadmin.py?colID=%s&ln=%s&mtype=perform_addcollection">Create new collection</a></small></td>
<td>2. <small><a href="%s/admin/websearch/websearchadmin.py?colID=%s&ln=%s&mtype=perform_addcollectiontotree">Attach collection to tree</a></small></td>
text += """</td><td></td><td></td><td></td><td><table border="0" cellspacing="0" cellpadding="0"><tr><td>
"""
if i == 0:
tstack.append((id_son, dad, 1))
else:
tstack.append((id_son, dad, tables))
if up == 1 and edit:
- text += """<a href="%s/admin/websearch/websearchadmin.py/modifycollectiontree?colID=%s&ln=%s&move_up=%s&rtype=%s#%s"><img border="0" src="%s/img/smallup.gif" title="Move collection up"></a>""" % (weburl, colID, ln, i, rtype, tree[i][0], weburl)
+ text += """<a href="%s/admin/websearch/websearchadmin.py/modifycollectiontree?colID=%s&ln=%s&move_up=%s&rtype=%s#%s"><img border="0" src="%s/img/smallup.gif" title="Move collection up"></a>""" % (CFG_SITE_URL, colID, ln, i, rtype, tree[i][0], CFG_SITE_URL)
else:
text += """ """
text += "</td><td>"
if down == 1 and edit:
- text += """<a href="%s/admin/websearch/websearchadmin.py/modifycollectiontree?colID=%s&ln=%s&move_down=%s&rtype=%s#%s"><img border="0" src="%s/img/smalldown.gif" title="Move collection down"></a>""" % (weburl, colID, ln, i, rtype, tree[i][0], weburl)
+ text += """<a href="%s/admin/websearch/websearchadmin.py/modifycollectiontree?colID=%s&ln=%s&move_down=%s&rtype=%s#%s"><img border="0" src="%s/img/smalldown.gif" title="Move collection down"></a>""" % (CFG_SITE_URL, colID, ln, i, rtype, tree[i][0], CFG_SITE_URL)
else:
text += """ """
text += "</td><td>"
if edit:
if move_from and move_to:
tmove_from = move_from
move_from = ''
if not (move_from == "" and i == 0) and not (move_from != "" and int(move_from[1:len(move_from)]) == i and rtype == move_from[0]):
check = "true"
if move_from:
#if tree_from[move_from_id][0] == tree_to[i][0] or not check_col(tree_to[i][0], tree_from[move_from_id][0]):
# check = ''
#elif not check_col(tree_to[i][0], tree_from[move_from_id][0]):
# check = ''
#if not check and (tree_to[i][0] == 1 and tree_from[move_from_id][3] == tree_to[i][0] and move_from_rtype != rtype):
# check = "true"
if check:
text += """<a href="%s/admin/websearch/websearchadmin.py/modifycollectiontree?colID=%s&ln=%s&move_from=%s&move_to=%s%s&rtype=%s#tree"><img border="0" src="%s/img/move_to.gif" title="Move '%s' to '%s'"></a>
<dd>When deleting a collection, you also deletes all data related to the collection like translations, relations to other collections and information about which rank methods to use.
<br />For more information, please go to the <a title="See guide" href="%s/help/admin/websearch-admin-guide">WebSearch guide</a> and read the section regarding deleting a collection.</dd>
</dl>
</strong>
</span>
- """ % weburl
+ """ % CFG_SITE_URL
col_dict = dict(get_def_name('', "collection"))
if colID != 1 and colID and col_dict.has_key(int(colID)):
output = """<b><span class="info">Can not delete a collection that is a part of the collection tree, remove collection from the tree and try again.</span></b>"""
else:
subtitle = """4. Delete collection"""
output = """<b><span class="info">Not possible to delete the root collection</span></b>"""
<td>11. <small><a href="editcollection?colID=%s&ln=%s&mtype=perform_manage_external_collections#11">Configuration of related external collections</a></small></td>
<td>12. <small><a href="editcollection?colID=%s&ln=%s&mtype=perform_showdetailedrecordoptions#12">Detailed record page options</a></small></td>
if collection_table_update_time > collection_web_update_time:
output += """<br /><b><span class="info">Warning: The collections has been modified since last time Webcoll was executed, to process the changes, Webcoll must be executed.</span></b><br />"""
res = run_sql("select id, proc, host, user, runtime, sleeptime, arguments, status, progress from schTASK where proc='webcoll' and runtime< now() ORDER by runtime")
res = run_sql("select id, proc, host, user, runtime, sleeptime, arguments, status, progress from schTASK where proc='bibindex' and runtime< now() ORDER by runtime")
res = run_sql("select id, proc, host, user, runtime, sleeptime, arguments, status, progress from schTASK where proc='webcoll' and runtime > now() ORDER by runtime")
res = run_sql("select id, proc, host, user, runtime, sleeptime, arguments, status, progress from schTASK where proc='bibindex' and runtime > now() ORDER by runtime")
output += """<br /><b><span class="info">Warning: Webcoll is not scheduled for a future run by bibsched, any updates to the collection will not be processed.</span></b><br />"""
if bibindex_future == "":
output += """<br /><b><span class="info">Warning: Bibindex is not scheduled for a future run by bibsched, any updates to the records will not be processed.</span></b><br />"""
tree = tree[0:ssize] + ntree + tree[ssize:len(tree)]
return tree
except StandardError, e:
return ()
def add_col_dad_son(add_dad, add_son, rtype):
"""Add a son to a collection (dad)
add_dad - add to this collection id
add_son - add this collection id
rtype - either regular or virtual"""
try:
res = run_sql("SELECT score FROM collection_collection WHERE id_dad=%s ORDER BY score ASC" % add_dad)
highscore = 0
for score in res:
if int(score[0]) > highscore:
highscore = int(score[0])
highscore += 1
res = run_sql("INSERT INTO collection_collection(id_dad,id_son,score,type) values(%s,%s,%s,'%s')" % (add_dad, add_son, highscore, rtype))
return (1, highscore)
except StandardError, e:
return (0, e)
def compare_on_val(first, second):
"""Compare the two values"""
return cmp(first[1], second[1])
def get_col_fld(colID=-1, type = '', id_field=''):
"""Returns either all portalboxes associated with a collection, or based on either colID or language or both.
colID - collection id
ln - language id"""
sql = "SELECT id_field,id_fieldvalue,type,score,score_fieldvalue FROM collection_field_fieldvalue, field WHERE id_field=field.id"
try:
if colID > -1:
sql += " AND id_collection=%s" % colID
if id_field:
sql += " AND id_field=%s" % id_field
if type:
sql += " AND type='%s'" % type
sql += " ORDER BY type, score desc, score_fieldvalue desc"
res = run_sql(sql)
return res
except StandardError, e:
return ""
def get_col_pbx(colID=-1, ln='', position = ''):
"""Returns either all portalboxes associated with a collection, or based on either colID or language or both.
colID - collection id
ln - language id"""
sql = "SELECT id_portalbox, id_collection, ln, score, position, title, body FROM collection_portalbox, portalbox WHERE id_portalbox = portalbox.id"
try:
if colID > -1:
sql += " AND id_collection=%s" % colID
if ln:
sql += " AND ln='%s'" % ln
if position:
sql += " AND position='%s'" % position
sql += " ORDER BY position, ln, score desc"
res = run_sql(sql)
return res
except StandardError, e:
return ""
def get_col_fmt(colID=-1):
"""Returns all formats currently associated with a collection, or for one specific collection
colID - the id of the collection"""
try:
if colID not in [-1, "-1"]:
res = run_sql("SELECT id_format, id_collection, code, score FROM collection_format, format WHERE id_format = format.id AND id_collection=%s ORDER BY score desc" % colID)
else:
res = run_sql("SELECT id_format, id_collection, code, score FROM collection_format, format WHERE id_format = format.id ORDER BY score desc")
return res
except StandardError, e:
return ""
def get_col_rnk(colID, ln):
""" Returns a list of the rank methods the given collection is attached to
colID - id from collection"""
try:
res1 = dict(run_sql("SELECT id_rnkMETHOD, '' FROM collection_rnkMETHOD WHERE id_collection=%s" % colID))
res2 = get_def_name('', "rnkMETHOD")
result = filter(lambda x: res1.has_key(x[0]), res2)
return result
except StandardError, e:
return ()
def get_pbx():
"""Returns all portalboxes"""
try:
res = run_sql("SELECT id, title, body FROM portalbox ORDER by title,body")
return res
except StandardError, e:
return ""
def get_fld_value(fldvID = ''):
"""Returns fieldvalue"""
try:
sql = "SELECT id, name, value FROM fieldvalue"
if fldvID:
sql += " WHERE id=%s" % fldvID
sql += " ORDER BY name"
res = run_sql(sql)
return res
except StandardError, e:
return ""
def get_pbx_pos():
"""Returns a list of all the positions for a portalbox"""
position = {}
position["rt"] = "Right Top"
position["lt"] = "Left Top"
position["te"] = "Title Epilog"
position["tp"] = "Title Prolog"
position["ne"] = "Narrow by coll epilog"
position["np"] = "Narrow by coll prolog"
return position
def get_sort_nametypes():
"""Return a list of the various translationnames for the fields"""
type = {}
type['soo'] = 'Sort options'
type['seo'] = 'Search options'
type['sew'] = 'Search within'
return type
def get_fmt_nametypes():
"""Return a list of the various translationnames for the output formats"""
type = []
type.append(('ln', 'Long name'))
return type
def get_fld_nametypes():
"""Return a list of the various translationnames for the fields"""
type = []
type.append(('ln', 'Long name'))
return type
def get_col_nametypes():
"""Return a list of the various translationnames for the collections"""
type = []
type.append(('ln', 'Long name'))
return type
def find_last(tree, start_son):
"""Find the previous collection in the tree with the same father as start_son"""
id_dad = tree[start_son][3]
while start_son > 0:
start_son -= 1
if tree[start_son][3] == id_dad:
return start_son
def find_next(tree, start_son):
"""Find the next collection in the tree with the same father as start_son"""
id_dad = tree[start_son][3]
while start_son < len(tree):
start_son += 1
if tree[start_son][3] == id_dad:
return start_son
def remove_col_subcol(id_son, id_dad, type):
"""Remove a collection as a son of another collection in the tree, if collection isn't used elsewhere in the tree, remove all registered sons of the id_son.
id_son - collection id of son to remove
id_dad - the id of the dad"""
try:
if id_son != id_dad:
tree = get_col_tree(id_son)
res = run_sql("DELETE FROM collection_collection WHERE id_son=%s and id_dad=%s" % (id_son, id_dad))
else:
tree = get_col_tree(id_son, type)
res = run_sql("DELETE FROM collection_collection WHERE id_son=%s and id_dad=%s and type='%s'" % (id_son, id_dad, type))
if not run_sql("SELECT * from collection_collection WHERE id_son=%s and type='%s'" % (id_son, type)):
for (id, up, down, dad, rtype) in tree:
res = run_sql("DELETE FROM collection_collection WHERE id_son=%s and id_dad=%s" % (id, dad))
return (1, "")
except StandardError, e:
return (0, e)
def check_col(add_dad, add_son):
"""Check if the collection can be placed as a son of the dad without causing loops.
add_dad - collection id
add_son - collection id"""
try:
stack = [add_dad]
res = run_sql("SELECT id_dad FROM collection_collection WHERE id_dad=%s AND id_son=%s" % (add_dad, add_son))
if res:
raise StandardError
while len(stack) > 0:
colID = stack.pop()
res = run_sql("SELECT id_dad FROM collection_collection WHERE id_son=%s" % colID)
for id in res:
if int(id[0]) == int(add_son):
raise StandardError
else:
stack.append(id[0])
return (1, "")
except StandardError, e:
return (0, e)
def attach_rnk_col(colID, rnkID):
"""attach rank method to collection
rnkID - id from rnkMETHOD table
colID - id of collection, as in collection table """
try:
res = run_sql("INSERT INTO collection_rnkMETHOD(id_collection, id_rnkMETHOD) values (%s,%s)" % (colID, rnkID))
return (1, "")
except StandardError, e:
return (0, e)
def detach_rnk_col(colID, rnkID):
"""detach rank method from collection
rnkID - id from rnkMETHOD table
colID - id of collection, as in collection table """
try:
res = run_sql("DELETE FROM collection_rnkMETHOD WHERE id_collection=%s AND id_rnkMETHOD=%s" % (colID, rnkID))
return (1, "")
except StandardError, e:
return (0, e)
def switch_col_treescore(col_1, col_2):
try:
res1 = run_sql("SELECT score FROM collection_collection WHERE id_dad=%s and id_son=%s" % (col_1[3], col_1[0]))
res2 = run_sql("SELECT score FROM collection_collection WHERE id_dad=%s and id_son=%s" % (col_2[3], col_2[0]))
res = run_sql("UPDATE collection_collection SET score=%s WHERE id_dad=%s and id_son=%s" % (res2[0][0], col_1[3], col_1[0]))
res = run_sql("UPDATE collection_collection SET score=%s WHERE id_dad=%s and id_son=%s" % (res1[0][0], col_2[3], col_2[0]))
score - decides which portalbox is the most important
position - position on page the portalbox should appear."""
try:
if score:
res = run_sql("INSERT INTO collection_portalbox(id_portalbox, id_collection, ln, score, position) values (%s,%s,'%s',%s,'%s')" % (pbxID, colID, ln, score, position))
else:
res = run_sql("SELECT score FROM collection_portalbox WHERE id_collection=%s and ln='%s' and position='%s' ORDER BY score desc, ln, position" % (colID, ln, position))
if res:
score = int(res[0][0])
else:
score = 0
res = run_sql("INSERT INTO collection_portalbox(id_portalbox, id_collection, ln, score, position) values (%s,%s,'%s',%s,'%s')" % (pbxID, colID, ln, (score + 1), position))
return (1, "")
except StandardError, e:
return (0, e)
def add_col_fmt(colID, fmtID, score=''):
"""Add a output format to the collection.
colID - the id of the collection involved
fmtID - the id of the format.
score - the score of the format, decides sorting, if not given, place the format on top"""
try:
if score:
res = run_sql("INSERT INTO collection_format(id_format, id_collection, score) values (%s,%s,%s)" % (fmtID, colID, score))
else:
res = run_sql("SELECT score FROM collection_format WHERE id_collection=%s ORDER BY score desc" % colID)
if res:
score = int(res[0][0])
else:
score = 0
res = run_sql("INSERT INTO collection_format(id_format, id_collection, score) values (%s,%s,%s)" % (fmtID, colID, (score + 1)))
return (1, "")
except StandardError, e:
return (0, e)
def add_col_fld(colID, fldID, type, fldvID=''):
"""Add a sort/search/field to the collection.
colID - the id of the collection involved
fldID - the id of the field.
fldvID - the id of the fieldvalue.
type - which type, seo, sew...
score - the score of the format, decides sorting, if not given, place the format on top"""
try:
if fldvID and fldvID not in [-1, "-1"]:
run_sql("DELETE FROM collection_field_fieldvalue WHERE id_collection=%s AND id_field=%s and type='%s' and id_fieldvalue is NULL" % (colID, fldID, type))
res = run_sql("SELECT score FROM collection_field_fieldvalue WHERE id_collection=%s AND id_field=%s and type='%s' ORDER BY score desc" % (colID, fldID, type))
if res:
score = int(res[0][0])
res = run_sql("SELECT score_fieldvalue FROM collection_field_fieldvalue WHERE id_collection=%s AND id_field=%s and type='%s' ORDER BY score_fieldvalue desc" % (colID, fldID, type))
else:
res = run_sql("SELECT score FROM collection_field_fieldvalue WHERE id_collection=%s and type='%s' ORDER BY score desc" % (colID, type))
if res:
score = int(res[0][0]) + 1
else:
score = 1
res = run_sql("SELECT * FROM collection_field_fieldvalue where id_field=%s and id_collection=%s and type='%s' and id_fieldvalue=%s" % (fldID, colID, type, fldvID))
if not res:
run_sql("UPDATE collection_field_fieldvalue SET score_fieldvalue=score_fieldvalue+1 WHERE id_field=%s AND id_collection=%s and type='%s'" % (fldID, colID, type))
res = run_sql("INSERT INTO collection_field_fieldvalue(id_field, id_fieldvalue, id_collection, type, score, score_fieldvalue) values (%s,%s,%s,'%s',%s,%s)" % (fldID, fldvID, colID, type, score, 1))
else:
return (0, (1, "Already exists"))
else:
res = run_sql("SELECT * FROM collection_field_fieldvalue WHERE id_collection=%s AND type='%s' and id_field=%s and id_fieldvalue is NULL" % (colID, type, fldID))
if res:
return (0, (1, "Already exists"))
else:
run_sql("UPDATE collection_field_fieldvalue SET score=score+1")
res = run_sql("INSERT INTO collection_field_fieldvalue(id_field, id_collection, type, score,score_fieldvalue) values (%s,%s,'%s',%s, 0)" % (fldID, colID, type, 1))
return (1, "")
except StandardError, e:
return (0, e)
def modify_dbquery(colID, dbquery=None):
"""Modify the dbquery of an collection.
colID - the id of the collection involved
dbquery - the new dbquery"""
# BTW, sometimes '' is passed instead of None, so change it to None
if not dbquery:
dbquery = None
try:
res = run_sql("UPDATE collection SET dbquery=%s WHERE id=%s", (dbquery, colID))
pbxID - the id of the portalbox that should be modified
sel_ln - the language of the portalbox that should be modified
title - the title
body - the content
score - if several portalboxes in one position, who should appear on top.
position - position on page"""
try:
if title:
res = run_sql("UPDATE portalbox SET title=%s WHERE id=%s", (title, pbxID))
if body:
res = run_sql("UPDATE portalbox SET body=%s WHERE id=%s", (body, pbxID))
if score:
res = run_sql("UPDATE collection_portalbox SET score='%s' WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (score, colID, pbxID, sel_ln))
if position:
res = run_sql("UPDATE collection_portalbox SET position='%s' WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (position, colID, pbxID, sel_ln))
return (1, "")
except Exception, e:
return (0, e)
def switch_fld_score(colID, id_1, id_2):
"""Switch the scores of id_1 and id_2 in collection_field_fieldvalue
colID - collection the id_1 or id_2 is connected to
id_1/id_2 - id field from tables like format..portalbox...
table - name of the table"""
try:
res1 = run_sql("SELECT score FROM collection_field_fieldvalue WHERE id_collection=%s and id_field=%s" % (colID, id_1))
res2 = run_sql("SELECT score FROM collection_field_fieldvalue WHERE id_collection=%s and id_field=%s" % (colID, id_2))
if res1[0][0] == res2[0][0]:
return (0, (1, "Cannot rearrange the selected fields, either rearrange by name or use the mySQL client to fix the problem."))
else:
res = run_sql("UPDATE collection_field_fieldvalue SET score=%s WHERE id_collection=%s and id_field=%s" % (res2[0][0], colID, id_1))
res = run_sql("UPDATE collection_field_fieldvalue SET score=%s WHERE id_collection=%s and id_field=%s" % (res1[0][0], colID, id_2))
colID - collection the id_1 or id_2 is connected to
id_1/id_2 - id field from tables like format..portalbox...
table - name of the table"""
try:
res1 = run_sql("SELECT score_fieldvalue FROM collection_field_fieldvalue WHERE id_collection=%s and id_field=%s and id_fieldvalue=%s" % (colID, id_1, fldvID_1))
res2 = run_sql("SELECT score_fieldvalue FROM collection_field_fieldvalue WHERE id_collection=%s and id_field=%s and id_fieldvalue=%s" % (colID, id_1, fldvID_2))
if res1[0][0] == res2[0][0]:
return (0, (1, "Cannot rearrange the selected fields, either rearrange by name or use the mySQL client to fix the problem."))
else:
res = run_sql("UPDATE collection_field_fieldvalue SET score_fieldvalue=%s WHERE id_collection=%s and id_field=%s and id_fieldvalue=%s" % (res2[0][0], colID, id_1, fldvID_1))
res = run_sql("UPDATE collection_field_fieldvalue SET score_fieldvalue=%s WHERE id_collection=%s and id_field=%s and id_fieldvalue=%s" % (res1[0][0], colID, id_1, fldvID_2))
return (1, "")
except Exception, e:
return (0, e)
def switch_pbx_score(colID, id_1, id_2, sel_ln):
"""Switch the scores of id_1 and id_2 in the table given by the argument.
colID - collection the id_1 or id_2 is connected to
id_1/id_2 - id field from tables like format..portalbox...
table - name of the table"""
try:
res1 = run_sql("SELECT score FROM collection_portalbox WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (colID, id_1, sel_ln))
res2 = run_sql("SELECT score FROM collection_portalbox WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (colID, id_2, sel_ln))
if res1[0][0] == res2[0][0]:
return (0, (1, "Cannot rearrange the selected fields, either rearrange by name or use the mySQL client to fix the problem."))
res = run_sql("UPDATE collection_portalbox SET score=%s WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (res2[0][0], colID, id_1, sel_ln))
res = run_sql("UPDATE collection_portalbox SET score=%s WHERE id_collection=%s and id_portalbox=%s and ln='%s'" % (res1[0][0], colID, id_2, sel_ln))
return (1, "")
except Exception, e:
return (0, e)
def switch_score(colID, id_1, id_2, table):
"""Switch the scores of id_1 and id_2 in the table given by the argument.
colID - collection the id_1 or id_2 is connected to
id_1/id_2 - id field from tables like format..portalbox...
table - name of the table"""
try:
res1 = run_sql("SELECT score FROM collection_%s WHERE id_collection=%s and id_%s=%s" % (table, colID, table, id_1))
res2 = run_sql("SELECT score FROM collection_%s WHERE id_collection=%s and id_%s=%s" % (table, colID, table, id_2))
if res1[0][0] == res2[0][0]:
return (0, (1, "Cannot rearrange the selected fields, either rearrange by name or use the mySQL client to fix the problem."))
res = run_sql("UPDATE collection_%s SET score=%s WHERE id_collection=%s and id_%s=%s" % (table, res2[0][0], colID, table, id_1))
res = run_sql("UPDATE collection_%s SET score=%s WHERE id_collection=%s and id_%s=%s" % (table, res1[0][0], colID, table, id_2))
datalist=["""You are not a user authorized to perform admin tasks, try to
- <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, weburl)]),
+ <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, CFG_SITE_URL)]),
datalist=["""You are not a user authorized to perform admin tasks, try to
- <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, weburl)]),
+ <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, CFG_SITE_URL)]),
datalist=["""You are not a user authorized to perform admin tasks, try to
- <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, weburl)]),
+ <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, CFG_SITE_URL)]),
datalist=["""You are not a user authorized to perform admin tasks, try to
- <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, weburl)]),
+ <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, CFG_SITE_URL)]),
datalist=["""You are not a user authorized to perform admin tasks, try to
- <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, weburl)]),
+ <a href="%s/youraccount/login?referer=%s/admin/websearch/websearchadmin.py/">login</a> with another account.""" % (CFG_SITE_SECURE_URL, CFG_SITE_URL)]),
(re.compile(r"/search"), "<p>" + _("This collection is restricted. If you think you have right to access it, please authenticate yourself.") + "</p>"),
)
msg = ""
for regexp, txt in login_referrer2msg:
if regexp.search(referer):
msg = txt
break
# FIXME: Temporary Hack to help CDS current migration
if CFG_CERN_SITE and apache_msg:
return msg + apache_msg
if apache_msg:
msg += apache_msg + "<p>Otherwise please, provide the correct authorization" \
" data in the following form.</p>"
internal = None
for system in CFG_EXTERNAL_AUTHENTICATION.keys():
if not CFG_EXTERNAL_AUTHENTICATION[system][0]:
internal = system
break
register_available = CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS <= 1 and internal
methods = CFG_EXTERNAL_AUTHENTICATION.keys()
methods.sort()
selected = ''
for method in methods:
if CFG_EXTERNAL_AUTHENTICATION[method][1]:
selected = method
break
return websession_templates.tmpl_login_form(
ln = ln,
referer = referer,
internal = internal,
register_available = register_available,
methods = methods,
selected_method = selected,
msg = msg,
)
# perform_logout: display the message of not longer authorized,
Displays a form for the user to ask for his password sent by email.
Parameters:
- 'ln' *string* - The language to display the interface in
- 'msg' *string* - Explicative message on top of the form.
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<p>" + _("If you have lost the password for your %(sitename)s %(x_fmt_open)sinternal account%(x_fmt_close)s, then please enter your email address in the following form in order to have a password reset link emailed to you.") % {'x_fmt_open' : '<em>', 'x_fmt_close' : '</em>', 'sitename' : CFG_SITE_NAME_INTL[ln]} + "</p>"
out += "<p>" + _("If you have been using the %(x_fmt_open)sCERN login system%(x_fmt_close)s, then you can recover your password through the %(x_url_open)sCERN authentication system%(x_url_close)s.") % {'x_fmt_open' : '<em>', 'x_fmt_close' : '</em>', 'x_url_open' : '<a href="https://cern.ch/lightweightregistration/ResetPassword.aspx%s">' \
out += "<p>" + _("Note that if you have been using an external login system, then we cannot do anything and you have to ask there.") + " "
out += _("Alternatively, you can ask %s to change your login system from external to internal.") % ("""<a href="mailto:%(email)s">%(email)s</a>""" % { 'email' : CFG_SITE_SUPPORT_EMAIL }) + "</p>"
- 'ln' *string* - The language to display the interface in
- 'uid' *string* - The user id
- 'guest' *boolean* - If the user is guest
- 'CFG_CERN_SITE' *boolean* - If the site is a CERN site
"""
# load the right message language
_ = gettext_set_language(ln)
out = """<p>%(account_offer)s</p>
<blockquote>
<dl>
""" % {
'account_offer' : _("%s offers you the possibility to personalize the interface, to set up your own personal library of documents, or to set up an automatic alert query that would run periodically and would notify you of search results by email.") % CFG_SITE_NAME_INTL[ln],
}
if not guest:
out += """
<dt>
<a href="./edit?ln=%(ln)s">%(your_settings)s</a>
</dt>
<dd>%(change_account)s</dd>""" % {
'ln' : ln,
'your_settings' : _("Your Settings"),
'change_account' : _("Set or change your account email address or password. Specify your preferences about the look and feel of the interface.")
'search_explain' : _("View all the searches you performed during the last 30 days."),
'your_baskets' : _("Your Baskets"),
'basket_explain' : _("With baskets you can define specific collections of items, store interesting records you want to access later or share with others."),
}
if guest:
out += self.tmpl_warning_guest_user(ln = ln, type = "baskets")
'explain_alerts' : _("Subscribe to a search which will be run periodically by our service. The result can be sent to you via Email or stored in one of your baskets."),
}
if guest:
out += self.tmpl_warning_guest_user(type="alerts", ln = ln)
'explain_loans' : _("Check out book you have on loan, submit borrowing requests, etc. Requires CERN ID."),
}
out += """
</dl>
</blockquote>"""
return out
def tmpl_warning_guest_user(self, ln, type):
"""
Displays a warning message about the specified type
Parameters:
- 'ln' *string* - The language to display the interface in
- 'type' *string* - The type of data that will get lost in case of guest account (for the moment: 'alerts' or 'baskets')
"""
# load the right message language
_ = gettext_set_language(ln)
if (type=='baskets'):
msg = _("You are logged in as a guest user, so your baskets will disappear at the end of the current session.") + ' '
elif (type=='alerts'):
msg = _("You are logged in as a guest user, so your alerts will disappear at the end of the current session.") + ' '
msg += _("If you wish you can %(x_url_open)slogin or register here%(x_url_close)s.") % {'x_url_open': '<a href="' + CFG_SITE_SECURE_URL + '/youraccount/login?ln=' + ln + '">',
'x_url_close': '</a>'}
return """<table class="errorbox" summary="">
<tr>
<th class="errorboxheader">%s</th>
</tr>
</table>""" % msg
def tmpl_account_body(self, ln, user):
"""
Displays the body of the actions of the user
Parameters:
- 'ln' *string* - The language to display the interface in
- 'user' *string* - The username (nickname or email)
"""
# load the right message language
_ = gettext_set_language(ln)
out = _("You are logged in as %(x_user)s. You may want to a) %(x_url1_open)slogout%(x_url1_close)s; b) edit your %(x_url2_open)saccount settings%(x_url2_close)s.") %\
'intro2' : _("If you want to complete this account registration, please go to:"),
'link' : "%s/youraccount/access%s" %
(CFG_SITE_SECURE_URL, make_canonical_urlargd({
'ln' : ln,
'mailcookie' : address_activation_key
}, {})),
'outro' : _("in order to confirm the validity of this request."),
'outro2' : _("Please note that this URL will remain valid for about %(days)s days only.") % {'days' : CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS},
}
return out
def tmpl_account_emailSent(self, ln, email):
"""
Displays a confirmation message for an email sent
Parameters:
- 'ln' *string* - The language to display the interface in
- 'email' *string* - The email to which the message has been sent
"""
# load the right message language
_ = gettext_set_language(ln)
out =""
out += _("Okay, a password reset link has been emailed to %s.") % email
return out
def tmpl_account_delete(self, ln):
"""
Displays a confirmation message about deleting the account
Parameters:
- 'ln' *string* - The language to display the interface in
"""
# load the right message language
_ = gettext_set_language(ln)
out = "<p>" + _("""Deleting your account""") + '</p>'
return out
def tmpl_account_logout(self, ln):
"""
Displays a confirmation message about logging out
Parameters:
- 'ln' *string* - The language to display the interface in
"""
# load the right message language
_ = gettext_set_language(ln)
out = _("You are no longer recognized by our system.") + ' '
if CFG_EXTERNAL_AUTH_USING_SSO and CFG_EXTERNAL_AUTH_LOGOUT_SSO:
out += _("""You are still recognized by the centralized
%(x_fmt_open)sSSO%(x_fmt_close)s system. You can
%(x_url_open)slogout from SSO%(x_url_close)s, too.""") % \
- 'ln' *string* - The language to display the interface in
- 'referer' *string* - The referer URL - will be redirected upon after login
- 'internal' *boolean* - If we are producing an internal authentication
- 'register_available' *boolean* - If users can register freely in the system
- 'methods' *array* - The available authentication methods
- 'selected_method' *string* - The default authentication method
- 'msg' *string* - The message to print before the form, if needed
"""
# load the right message language
_ = gettext_set_language(ln)
if msg is "":
out = "<p>%(please_login)s" % {
'please_login' : _("If you already have an account, please login using the form below.")
}
if CFG_CERN_SITE:
out += "<p>" + _("If you don't own a CERN account yet, you can register a %(x_url_open)snew CERN lightweight account%(x_url_close)s.") % {'x_url_open' : '<a href="https://www.cern.ch/lightweightregistration/RegisterAccount.aspx">', 'x_url_close' : '</a>'} + "</p>"
else:
if register_available:
out += "<p>"+_("If you don't own an account yet, please %(x_url_open)sregister%(x_url_close)s an internal account.") %\
out += "<p>" + _("It is not possible to create an account yourself. Contact %s if you want an account.") % ('<a href="mailto:%s">%s</a>' % (CFG_SITE_SUPPORT_EMAIL, CFG_SITE_SUPPORT_EMAIL)) + "</p>"
else:
out = "<p>%s</p>" % msg
out += """<form method="post" action="../youraccount/login">
'password_contain' : _("The password phrase may contain punctuation, spaces, etc."),
'retype' : _("Retype Password"),
'register' : _("register"),
'explain_acc' : _("Please do not use valuable passwords such as your Unix, AFS or NICE passwords with this service. Your email address will stay strictly confidential and will not be disclosed to any third party. It will be used to identify you for personal services of %s. For example, you may set up an automatic alert search that will look for new preprints and will notify you daily of new arrivals by email.") % CFG_SITE_NAME,
@param nb_admin_group: number of groups the user is admin of
@param nb_member_group: number of groups the user is member of
@param total_group: number of groups the user belongs to
@param ln: language
return: html output.
"""
_ = gettext_set_language(ln)
out = _("You can consult the list of %(x_url_open)s%(x_nb_total)i groups%(x_url_close)s you are subscribed to (%(x_nb_member)i) or administering (%(x_nb_admin)i).")
return page(title=_("Role authorization request"), req=req, body=_("This request for an authorization has already been authorized."), uid=webuser.getUid(req), navmenuid='youraccount', language=args['ln'])
return page(title=title,
body=webaccount.perform_back(
_("You have successfully obtained an authorization as %(x_role)s! "
"This authorization will last until %(x_expiration)s and until "
"you close your browser if you are a guest user.") %
body = "<p>" + _("You have confirmed the validity of your email"
" address!") + "</p>"
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS == 1:
body += "<p>" + _("Please, wait for the administrator to "
"enable your account.") + "</p>"
else:
uid = webuser.update_Uid(req, email)
body += "<p>" + _("You can now go to %(x_url_open)syour account page%(x_url_close)s.") % {'x_url_open' : '<a href="/youraccount/display?ln=%s">' % args['ln'], 'x_url_close' : '</a>'} + "</p>"
body = "<p>" + _("You have already confirmed the validity of your email address!") + "</p>"
if CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS == 1:
body += "<p>" + _("Please, wait for the administrator to "
"enable your account.") + "</p>"
else:
body += "<p>" + _("You can now go to %(x_url_open)syour account page%(x_url_close)s.") % {'x_url_open' : '<a href="/youraccount/display?ln=%s">' % args['ln'], 'x_url_close' : '</a>'} + "</p>"
mess = _("Desired nickname %s already exists in the database.") % args['p_nickname']
mess += " " + _("Please try again.")
act = "register"
title = _("Registration failure")
elif ruid == 5:
mess = _("Users cannot register themselves, only admin can register them.")
act = "register"
title = _("Registration failure")
elif ruid == 6:
mess = _("The site is having troubles in sending you an email for confirming your email address.") + _("The error has been logged and will be taken in consideration as soon as possibile.")
act = "register"
title = _("Registration failure")
else:
# this should never happen
mess = _("Internal Error")
act = "register"
title = _("Registration failure")
return page(title=title,
body=webaccount.perform_back(mess,act, (act == 'register' and _("register") or ""), args['ln']),
page_body += '<p>' + (_("You may want to start browsing from %(rooturl)s or have a look at the %(x_url_open)s index of the %(category)s pages%(x_url_close)s.") % \
- <span class="guideheader">T</span>o add a document type to WebSubmit, you should go to the <a target=top href="<WEBURL>/admin/websubmit/index.php">main page</a>
+ <span class="guideheader">T</span>o add a document type to WebSubmit, you should go to the <a target=top href="<CFG_SITE_URL>/admin/websubmit/index.php">main page</a>
and click on "New Doctype" in the left blue panel.<br/><br/>
<span class="guideheader">E</span>ven once created, a document type will not appear automatically on this page. To configure the list of catalogues and document
- types displayed on this page, the administrator shall go to the <a target=top href="<WEBURL>/admin/websubmit/editCatalogues.php">edit catalogues</a>
+ types displayed on this page, the administrator shall go to the <a target=top href="<CFG_SITE_URL>/admin/websubmit/editCatalogues.php">edit catalogues</a>
page. (see the <a href="#catalogues">guide section</a>)<br clear="all"/>
<h3>The user can then click on the document type he is interested in.</h3>
- <IMG src="<WEBURL>/img/admin/websubmit-admin-guide-menu_doc.png" alt="Document type Page" class="guideimg" align="left">
+ <IMG src="<CFG_SITE_URL>/img/admin/websubmit-admin-guide-menu_doc.png" alt="Document type Page" class="guideimg" align="left">
<span class="guideheader">T</span>he text appearing under the header containing the name of the document
can be configured by going to the <a target=top href="websubmit-admin">main page</a>, click on
the title of the document type then on the "Edit Document Types Details" button.<br/><br/>
<span class="guideheader">Y</span>ou can associate several categories to a document type which can be defined by going to the
<a target=top href="websubmit-admin">main page</a>, click on the title of the document type
then on the "View Categories" button. The selected category will be saved in a file named "comboXXX"
(where XXX is the short name of the document type) in the submission directory.<br/><br/>
<span class="guideheader">T</span>o add an action button to this page, first implement this action by going to the
<a target=top href="websubmit-admin">main page</a>, click on the title of the document type then
on the "Add a new submission" button. If the action is already implemented and the button still does not appear
on the submision page, then you should edit the details of this implementation: go to the
<a target=top href="websubmit-admin">main page</a>, click on the title of the document type then
on the icon in the "Edit Submission" column and in the line of the desired action. There you should set the
"Displayed" form field to "YES".<br/><br/>
<span class="guideheader">Y</span>ou can also change the order of the buttons, by going to the <a target=top href="websubmit-admin">
main page</a>, click on the title of the document type then on the icon in the "Edit Submission" column and in the
line of the desired action. There you can set the "buttonorder" form field.<br/><br clear="all"/>
<h3>The user now may choose a category, then click on the action button he wishes.<br/>The submission starts, the first page of the web form appears.</h3>
- <IMG src="<WEBURL>/img/admin/websubmit-admin-guide-form.png" alt="Document type Page" class="guideimg" align="left">
+ <IMG src="<CFG_SITE_URL>/img/admin/websubmit-admin-guide-form.png" alt="Document type Page" class="guideimg" align="left">
<span class="guideheader">T</span>his web form is composed of several pages, on each of these
pages form fields can be found. To modify the number of pages, add or withdraw form fields and modify
the texts before each form field, you shall go to the <a target=top href="websubmit-admin">main page</a>,
click on the title of the document type then on the icon in the "Edit Submission Pages" column and in the line of the
desired action. (see the <a href="#actionimplement">guide section</a>)<br/><br clear="all"/>
<h3>On the last page of the submission, there should be a button like in the following image which will
trigger the end script</h3>
- <IMG src="<WEBURL>/img/admin/websubmit-admin-guide-end_action.png" alt="Document type End Page" class="guideimg" align="left">
+ <IMG src="<CFG_SITE_URL>/img/admin/websubmit-admin-guide-end_action.png" alt="Document type End Page" class="guideimg" align="left">
<span class="guideheader">T</span>his button is defined like any other form field. Its definition should include
a <i> onclick="finish();"</i> javascript attribute.<br/><br/>
<span class="guideheader">A</span>fter clicking this button, WebSubmit will apply the end script functions
to the gathered data. To modify the end script, you shall go to the <a target=top href="websubmit-admin">
main page</a>, click on the title of the document type then on the icon in the "Edit Functions" column and in the line
of the desired action. (see the <a href="#implementfunctions">guide section</a>)<br clear="all"/>
This function searches for the document in the database and stores the recid of this document in the "SN" file and in a global variable "sysno".<br/>
The function conducts the search based upon the document's report-number (and relies upon the global variable "rn") so the "Get_Report_Number" function should be called before this one.<br/>
<i>This function replaces the older function "Get_Sysno".</i><br/>
This value depends on the web form configuration you did. It should contain the name of the form element used for storing the reference of the document.
If the authentication module (login) is active in webSubmit, this function compares the current login with the email of the original submitter. If it is the same (or if the current user has superuser rights), we go on. If it differs, an error message is issued.
indicates the file in which the program will find the counter for this reference generation.<br/>
The value of this parameter may contain one of:<br/>
"<b><PA>categ</PA></b>": in this case this string is replaced with the content of the file [altrnin]<br/>
"<b><PA>yy</PA></b>": in this case this string is replaced by the current year (4 digits) if [altyeargen]
is set to "AUTO", or by the content of the [altyeargen] file in any other case. (this content should be formatted
as a date (dd/mm/yyyy).<br/>
"<b><PA>file:<i>name_of_file</i></PA></b>": in this case, this string is replaced by the first line of the given file<br />
"<b><PA>file*:<i>name_of_file</i></PA></b>": in this case, this string is replaced by all the lines of the given file, separated by a dash ('-') character.
This is the format used by the program to create the reference. The program computes the value of the
parameter and appends a "-" followed by the current value of the counter increased by 1.<br/>
The value of this parameter may contain one of:<br/>
"<b><PA>categ</PA></b>": in this case this string is replaced with the content of the file [altrnin]<br/>
"<b><PA>yy</PA></b>": in this case this string is replaced by the current year (4 digits) if [altyeargen]
is set to "AUTO", or by the content of the [altyeargen] file in any other case. (this content should be formatted
as a date (dd/mm/yyyy).
<br/>
"<b><PA>file:<i>name_of_file</i></PA></b>": in this case, this string is replaced by the first line of the given file<br />
"<b><PA>file*:<i>name_of_file</i></PA></b>": in this case, this string is replaced by all the lines of the given file, separated by a dash ('-') character.
</small></td>
</tr>
<tr>
<td valign="top"><small><b>rnin</b></small></td>
<td><small>
This parameter contains the name of the file in which the program will find the category if needed. The content
of thif file will then replace the string <PA>categ</PA> in the reference format or in the counter
email addresses of the people who will receive this email (comma separated list). this parameter may contain the <b><CATEG></b> string. In which case the variable computed from the [categformatDAM] parameter replaces this string.<br/>
contains a regular expression used to compute the category of the document given the reference of the document.<br/>
eg.: if [categformatAFP]="TEST-<CATEG>-.*" and the reference of the document is "TEST-CATEGORY1-2001-001", then the computed category equals "CATEGORY1"
email addresses of the people who will receive this email (comma separated list). this parameter may contain the <b><CATEG></b> string. In which case the variable computed from the [categformatDAM] parameter replaces this string.<br/>
raise InvenioWebSubmitFileError, "A file for docname '%s' for the recid '%s' already exists for the format '%s'" % (self.docname, self.recid, docfile.get_format())
email_txt += "Please note that the modifications will be taken into account in a couple of minutes.\n\nBest regards,\nThe %s Server support Team" % CFG_SITE_NAME
- message = "A revised version of document %s has been submitted.\n\nTitle: %s\nAuthor(s): %s\nURL: <%s/record/%s>%s" % (rn,title,author,weburl,sysno,note)
+ message = "A revised version of document %s has been submitted.\n\nTitle: %s\nAuthor(s): %s\nURL: <%s/record/%s>%s" % (rn,title,author,CFG_SITE_URL,sysno,note)
out += _("It has first been asked for refereing process on the ") + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br />'
out += _("Last request e-mail was sent to the publication committee chair on the ") + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br />'
if dates['dRefereeSel'] != None:
out += _("A referee has been selected by the publication committee on the ") + ' <strong class="headline">' + str(dates['dRefereeSel']) + '</strong><br />'
else:
out += _("No referee has been selected yet.")
if (status != "cancelled") and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="RefereeSel", linkText=_("Select a referee"))
out += '<br />'
if dates['dRefereeRecom'] != None:
out += _("The referee has sent his final recommendations to the publication committee on the ") + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br />'
else:
out += _("No recommendation from the referee yet.")
if (status != "cancelled") and (dates['dRefereeSel'] != None) and (isReferee == 0):
out += displaycplxdoc_displayauthaction (action="RefereeRecom", linkText=_("Send a recommendation"))
out += '<br />'
if dates['dPubComRecom'] != None:
out += _("The publication committee has sent his final recommendations to the project leader on the ") + ' <strong class="headline">' + str(dates['dPubComRecom']) + '</strong><br />'
else:
out += _("No recommendation from the publication committee yet.")
if (status != "cancelled") and (dates['dRefereeRecom'] != None) and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="PubComRecom", linkText=_("Send a recommendation"))
out += '<br />'
if status == "cancelled":
out += _("It has been cancelled by the author on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += _("It has been approved by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif status == "rejected":
out += _("It has been rejected by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
else:
out += _("No final decision taken yet.")
if (dates['dPubComRecom'] != None) and (isProjectLeader == 0):
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
out += '<br />'
elif apptype == "RPB":
out += _("It has first been asked for refereing process on the ") + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br />'
out += _("Last request e-mail was sent to the publication committee chair on the ") + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br />'
if dates['dEdBoardSel'] != None:
out += _("An editorial board has been selected by the publication committee on the ") + ' <strong class="headline">' + str(dates['dEdBoardSel']) + '</strong>'
if (status != "cancelled") and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="AddAuthorList", linkText=_("Add an author list"))
out += '<br />'
else:
out += _("No editorial board has been selected yet.")
if (status != "cancelled") and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="EdBoardSel", linkText=_("Select an editorial board"))
out += '<br />'
if dates['dRefereeSel'] != None:
out += _("A referee has been selected by the editorial board on the ") + ' <strong class="headline">' + str(dates['dRefereeSel']) + '</strong><br />'
else:
out += _("No referee has been selected yet.")
if (status != "cancelled") and (dates['dEdBoardSel'] != None) and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="RefereeSel", linkText=_("Select a referee"))
out += '<br />'
if dates['dRefereeRecom'] != None:
out += _("The referee has sent his final recommendations to the editorial board on the ") + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br />'
else:
out += _("No recommendation from the referee yet.")
if (status != "cancelled") and (dates['dRefereeSel'] != None) and (isReferee == 0):
out += displaycplxdoc_displayauthaction (action="RefereeRecom", linkText=_("Send a recommendation"))
out += '<br />'
if dates['dEdBoardRecom'] != None:
out += _("The editorial board has sent his final recommendations to the publication committee on the ") + ' <strong class="headline">' + str(dates['dRefereeRecom']) + '</strong><br />'
else:
out += _("No recommendation from the editorial board yet.")
if (status != "cancelled") and (dates['dRefereeRecom'] != None) and (isEdBoard == 0):
out += displaycplxdoc_displayauthaction (action="EdBoardRecom", linkText=_("Send a recommendation"))
out += '<br />'
if dates['dPubComRecom'] != None:
out += _("The publication committee has sent his final recommendations to the project leader on the ") + ' <strong class="headline">' + str(dates['dPubComRecom']) + '</strong><br />'
else:
out += _("No recommendation from the publication committee yet.")
if (status != "cancelled") and (dates['dEdBoardRecom'] != None) and (isPubCom == 0):
out += displaycplxdoc_displayauthaction (action="PubComRecom", linkText=_("Send a recommendation"))
out += '<br />'
if status == "cancelled":
out += _("It has been cancelled by the author on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += _("It has been approved by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif status == "rejected":
out += _("It has been rejected by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
else:
out += _("No final decision taken yet.")
if (dates['dPubComRecom'] != None) and (isProjectLeader == 0):
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
out += '<br />'
elif apptype == "RDA":
out += _("It has first been asked for refereing process on the ") + ' <strong class="headline">' + str(dates['dFirstReq']) + '</strong><br />'
out += _("Last request e-mail was sent to the project leader on the ") + ' <strong class="headline">' + str(dates['dLastReq']) + '</strong><br />'
if status == "cancelled":
out += _("It has been cancelled by the author on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif dates['dProjectLeaderAction'] != None:
if status == "approved":
out += _("It has been approved by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
elif status == "rejected":
out += _("It has been rejected by the project leader on the ") + ' <strong class="headline">' + str(dates['dProjectLeaderAction']) + '</strong><br />'
else:
out += _("No final decision taken yet.")
if isProjectLeader == 0:
out += displaycplxdoc_displayauthaction (action="ProjectLeaderDecision", linkText=_("Take a decision"))
if isAuthor == 0:
out += displaycplxdoc_displayauthaction (action="AuthorCancel", linkText=_("Cancel"))
'_("Unable to swap field at position %s with field at position %s on page %s of submission %s - could not move field at position %s to temporary field location")',
'_("Unable to swap field at position %s with field at position %s on page %s of submission %s - could not move field at position %s to position %s. Please ask Admin to check that a field was not stranded in a temporary position")',
'_("Unable to swap field at position %s with field at position %s on page %s of submission %s - could not move field that was located at position %s to position %s from temporary position. Field is now stranded in temporary position and must be corrected manually by an Admin")',
'_("Unable to move field at position %s to position %s on page %s of submission %s - could not decrement the position of the fields below position %s. Tried to recover - please check that field ordering is not broken")',
'_("Unable to move field at position %s to position %s on page %s of submission %s - could not increment the position of the fields at and below position %s. The field that was at position %s is now stranded in a temporary position.")',
'WRN_WEBSUBMITADMIN_TOOMANYROWS' : '_("Too many rows found for query [%s]. Expected %s, found %s.")',
'WRN_WEBSUBMITADMIN_NOROWSFOUND' : '_("No Rows found for query [%s].")',
## not warnings per-say, rather events that have taken place:
'WRN_WEBSUBMITADMIN_DELETED_FIELD_FROM_SUBMISSION_PAGE' : '_("Deleted field at position %s from page %s of submission \'%s\'")',
'WRN_WEBSUBMITADMIN_MOVED_FIELD_ON_SUBMISSION_PAGE' : '_("Moved field from position %s to position %s on page %s of submission \'%s\'")',
'WRN_WEBSUBMITADMIN_FIELDUPDATED' : '_("Updated details of field at position %s on page %s of submission \'%s\'")'
body_content += """<div><span class="info"><i>Really</i> remove document type "%s" and all of its configuration details?</span> <input name="doctypedeleteconfirm" class="adminbutton\""""\
body_content += """<div><span class="info"><i>Really</i> remove the Submission "%s" and all related details from Document Type "%s"?</span> <input name="doctypesubmissiondeleteconfirm" class="adminbutton" """ \
res = run_sql("SELECT lname FROM sbmCATEGORIES WHERE sname=%s and doctype=%s", (categ, doctype,))
if len(res) > 0:
categname = "Referee(s) for category: %s" % res[0][0]
else:
categname = "General Referee(s)"
t += '<tr><td colspan="2"><small><b>%s</b> </small></td></tr>' % categname
for referee in referees[role]:
if int(i/2) == i/2:
bgcolor = "#eeeeee"
else:
bgcolor = "#dddddd"
t += '<tr bgcolor="%s">' % bgcolor
t += '<td align="right"><small>'
t += referee[1]
t += '</small></td>'
t += '''<td><a href="" onclick="if (confirm('Are you sure you want to delete this referee?')){document.forms[0].todo.value='deleteuser';document.forms[0].id.value='%s';document.forms[0].role.value='%s';document.forms[0].submit();return false;}else{return false;}">''' % (referee[0], role)
t += '<img src="%s/img/iconcross.gif" border="0" alt="Remove" /></a>' % CFG_SITE_URL
t += '</td>'
t += '</tr>'
i += 1
# close table
t += "</table>"
return t
def displayAddUser(doctype):
"""Display controls for adding users"""
t = ""
# start displaying the table which will contain the add form
t += """
<table class="searchbox" summary="">
<tr>
<th class="portalboxheader">Add</th>
</tr>
<tr>
<td>
User:<br/>"""
users = list_registered_users()
if len(users) < 20:
numrows = len(users)
else:
numrows = 20
t += '<select multiple="multiple" name="addusers" size="%s">' % numrows
for user in users:
if user[1] != "":
t += '<option value="%s">%s</option>' % (user[0], user[1])
t += '</select><br/>'
t += '<select name="categ">'
t += '<option value="*">All categories</option>'
res = run_sql("select lname,sname FROM sbmCATEGORIES WHERE doctype=%s ORDER BY lname", (doctype,))
for row in res:
t += '<option value="%s">%s</option>' % (row[1], row[0])
t += '</select><br/>'
t += '''<input class="adminbutton" type="button" onclick="document.forms[0].todo.value='adduser';document.forms[0].submit();" value="ADD" />'''
res = run_sql("SELECT sbmSUBMISSIONS.* FROM sbmSUBMISSIONS,sbmACTION WHERE sactname=action and email=%s and id!='' "+docselect+" ORDER BY doctype,"+reqorder,(u_email,))
currentdoctype = ""
currentaction = ""
currentstatus = ""
submissions = []
for row in res:
if currentdoctype != row[1]:
currentdoctype = row[1]
currentaction = ""
currentstatus = ""
res2 = run_sql("SELECT ldocname FROM sbmDOCTYPE WHERE sdocname=%s",(currentdoctype,))
if res2:
ldocname = res2[0][0]
else:
ldocname = """***Unknown Document Type - (%s)""" % (currentdoctype,)
if currentaction != row[2]:
currentaction = row[2]
res2 = run_sql("SELECT lactname FROM sbmACTION WHERE sactname=%s",(currentaction,))