diff --git a/docs/admin/bibformat-admin-guide.rst b/docs/admin/bibformat-admin-guide.rst deleted file mode 100644 index 6ef70810f..000000000 --- a/docs/admin/bibformat-admin-guide.rst +++ /dev/null @@ -1,1720 +0,0 @@ -.. This file is part of Invenio - Copyright (C) 2014 CERN. - - Invenio is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License as - published by the Free Software Foundation; either version 2 of the - License, or (at your option) any later version. - - Invenio is distributed in the hope that it will be useful, but - WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - General Public License for more details. - - You should have received a copy of the GNU General Public License - along with Invenio; if not, write to the Free Software Foundation, Inc., - 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -.. _bibformat-admin-guide: - -BibFormat Admin Guide -===================== - -Contents --------- - -- **1. `Overview <#shortIntro>`__** - - - 1.1  \ `How BibFormat works <#philosophy>`__ - - 1.2  \ `Short Tutorial <#tutorial>`__ - - 1.3  \ `Administer Through the Web Interface or Through the - Configuration files <#administerWebFile>`__ - -- **2. `Configure Output Formats <#outputFormats>`__** - - - 2.1  \ `Add an Output Format <#addOutputFormat>`__ - - 2.2  \ `Remove an Output Format <#removeOutputFormat>`__ - - 2.3  \ `Edit the Rules of an Output Format <#rulesOutputFormat>`__ - - 2.4  \ `Edit the Attributes of an Output - Format <#attrsOutputFormat>`__ - - 2.5  \ `Check the Dependencies an Output - Format <#dependenciesOutputFormat>`__ - - 2.6  \ `Check the Validity an Output - Format <#validityOutputFormat>`__ - -- **3. `Configure Format Templates <#formatTemplates>`__** - - - 3.1  \ `Add a Format Template <#addFormatTemplate>`__ - - 3.2  \ `Remove a Format Template <#removeFormatTemplate>`__ - - 3.3  \ `Edit the Code of a Format - Template <#codeFormatTemplate>`__ - - 3.4  \ `Basic Editing <#editFormatTemplate>`__ - - 3.5  \ `Use Format Elements <#elementsInFormatTemplate>`__ - - 3.6  \ `Preview a Format Template <#previewFormatTemplate>`__ - - 3.7  \ `Internationalization - (i18n) <#internationalizationTemplate>`__ - - 3.8  \ `Escaping special HTML/XML - characters <#escapeFormatTemplate>`__ - - 3.9  \ `Edit the Attributes of a Format - Template <#attrsFormatTemplate>`__ - - 3.10 \ `Check the Dependencies of a Format - Template <#dependenciesFormatTemplate>`__ - - 3.11 \ `Check the Validity a Format - Template <#validityFormatTemplate>`__ - - 3.12 \ `XSL Format Templates <#xslFormatTemplate>`__ - -- **4. `Configure Format Elements <#FormatElements>`__** - - - 4.1  \ `Add a Format Element <#addFormatElement>`__ - - 4.2  \ `Remove a Format Element <#removeFormatElement>`__ - - 4.3  \ `Edit the Code of a Format Element <#codeFormatElement>`__ - - 4.4  \ `Preview a Format Element <#previewFormatElement>`__ - - 4.5  \ `Internationalization - (i18n) <#internationalizationFormatElement>`__ - - 4.6  \ `Escaping special HTML/XML - characters <#escapeFormatElement>`__ - - 4.7  \ `Edit the Attributes of a Format - Element <#attrsFormatElement>`__ - - 4.8  \ `Check the Dependencies of a Format - Element <#dependenciesFormatElement>`__ - - 4.9  \ `Check the Validity of a Format - Element <#validityFormatElement>`__ - - 4.10 \ `Browse the Format Elements - Documentation <#browseDocFormatElement>`__ - -- **5. `Run BibReformat <#BibReformat>`__** - - - 5.1  \ `Run BibReformat <#runBibReformat>`__ - -- **6. `Appendix <#Appendix>`__** - - - 6.1  \ `MARC Notation in Formats <#marcNotation>`__ - - 6.2  \ `Migrating from Previous BibFormat <#migration>`__ - - 6.3  \ `Integrating BibFormat into Dreamweaver - MX <#integrationDreamweaver>`__ - - 6.4  \ `FAQ <#faq>`__ - -1. Overview ------------ - -1.1 How BibFormat Works -~~~~~~~~~~~~~~~~~~~~~~~ - -BibFormat is in charge of formatting the bibliographic records that are -displayed to your users. It is called by the search engine when it has -to format a record. - -As you might need different kind of formatting depending on the type -of record, but potentially have a huge amount of records in your -database, you cannot specify for each of them how they should look. -Instead BibFormat uses a rule-based decision process to decide how to -format a record. - -The best way to understand how BibFormat works is to have a look at a -typical workflow: - -Step 1: - -|Use output format HD| - -When Invenio has to display a record, it asks BibFormat to format the -record with the given output format and language. For example here the -requested output format is hd, which is a short code for "HTML -Detailed". This means that somehow a user arrived on the page of the -record and asked for a detailed view of the record. - --------------- - -Step 2: - -1. Use Template [Picture HTML Detailed] if tag [980\_\_a] is equal to [PICTURE] -2. Use Template [Thesis HTML detailed] if tag [980\_\_a] is equal to [THESIS] -3. By default use [Default HTML Detailed]| - -|BibFormat Guide BFO HD rules| - -Beside is a screenshot of the "hd" or "HTML Detailed" output format. -You can see that the output format does not specify how to format the -record, but contains a set of rules which define which template must be -used. - -The rules are evaluated from top to bottom. Each rule defines a -condition on a field of the record, and a format template to use to -format the record if the condition matches. Let's say that the field -980\_\_a of the record is equal to "Picture". Then first rules matches, -and format template Picture HTML Detailed is used for formatting by -BibFormat. - -You can add, remove or edit output formats -`here `__ - --------------- - -Step 3: - -:: - -

-

-
- .- -

- -We see an extract of the Picture HTML Detailed format on the right, as -it is shown in the template editor. As you can see it is mainly written -using HTML. There are however some tags that are not part of standard -HTML. Those tags that starts with * tells BibFormat to write -the title of the record. We call these tags "elements". Some elements -have parameters. This is the case of the element, which -can take *separator* and *link* as parameters. The value of separator -will be used to separate authors' names and the link parameter tells if -links to authors' websites have to be created. All elements are -described in the `elements -documentation `__. - -You can add, remove or edit format templates -`here `__. - -In addition to this modified HTML language, BibFormat also supports XSL -stylesheets as format templates. Read the `XSL Format -Templates <#xslFormatTemplate>`__ section to learn more about XSLT -support for your format templates. - --------------- - -Step 4: - -:: - - def format_element(bfo, separator='; ', link='no'): -     """Prints the list of authors for the record - -     @param separator a character to separate the authors -     @param link if 'yes' print HTML links to authors     - """     - authors = bfo.fields("100__a")     - if link == 'yes':        - authors = map(lambda x: '' + x + '', authors)     - return authors.split(separator) - - -A format element is written in Python. It acts as a bridge between the -record in the database and the format template. Typically you will not -have to write or read format elements, just call them from the -templates. Each element outputs some text that is written in the -template where it is called. - -Developers can add new elements by creating a new file, naming it -with the name of element, and write a Python ``format_element`` function -that takes as parameters the parameters of the elements plus a special -one ``bfo``. Regular Python code can be used, including import of other -modules. - -In summary BibFormat is called by specifying a record and an output -format, which relies on different templates to do the formatting, and -which themselves rely on different format elements. Only developers need -to modify the format elements layer. - -Output Format - -Template - -Template - -Format Element - -Format Element - -Format Element - -Format Element - -You should now understand the philosophy behind BibFormat. - -1.2 Short Tutorial -~~~~~~~~~~~~~~~~~~ - -Let's try to create our own format. This format will just print the -title of a record. - -First go to the main `BibFormat admin -page `__. Then -click on the "Manage Ouput Format" links. You will see the list of all -output formats: - -|Output formats management page| - -This is were you can delete, create or check output formats. The menu -at the top of the page let you go to other admininistration pages. - -Click on the "Add New Output Format" button at the bottom of the -page. You can then fill in some attributes for the output format. Choose -"title" as code, "Only Title" as name and "Prints only title" as -description: - -|Screenshot of the Update Output Format Attributes page| - -Leave other fields blank, and click on the button "Update Output -format Attributes". - -You are then redirected to the rules editor. Notice the menu at the -top which let you close the editor, change the attributes again and -check the output format. However do not click on these links before -saving your modification of rules! - -|Output format menu| - -As our format does not need to have a different behaviour depending on -the record, we do not need to add new rules to the format. You just need -to select a format template in the "By default use" list. However we -first have to create our special format template that only print titles. -So close the editor using the menu at the top of the page, and in the -menu that just appeared instead, click on "Manage Format Templates". In -a similar way to output formats, you see the list of format templates. - -|Format template management page| - -Click on the "Add New Format Template" button at the bottom of the page. -As for the output format, fill in the attributes of the template with -name "Title" and any relevant description. - -|update format template attributes| - -Click on the "Update Output Format Attributes" button. You are -redirected to the template editor. The editor is divided in three parts. -The upper left part contains the code of the template. The bottom part -is a preview of the template. The part on the right side is a short -remainder of the format elements you can use in you template. You can -hide this documentation by clicking on "Hide Documentation". - -|Format template editor| - -The above screenshot shows the template code already filled in. It calls -the ``BFE_TITLE`` element. If you do not know the name of the element -you want to call, you can search for it using the embedded documentation -search. You can try to add other elements into your template, or write -some HTML formatting. - -When you are satisfied with your template, click on the save button, -close the editor and go back to the "Only titles" output format rules -editor. There select the template you have just created in the "Use by -default" menu and save the ouput format and you are done. - -This tutorial does not cover all aspects of the management of formats. -It also does not show all the power of output formats, as the one we -have created simply call a template. However you have seen enough to -configure BibFormat trough the web interface. Read the sections below to -learn more about it. - -1.3 Administer Through the Web Interface or Through the Configuration files -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -BibFormat can be administered in two ways. The first way is to use the -provided web interface. It should be the most convenient way of doing -for most users. The web interface is simple to use and provides great -tools to manage your formats. Its only limitation concerns the format -elements, which cannot be modified using it (But the web interface -provide a dynamically generated documentation of your elements). - -The other way to administer BibFormat is to directly modify the -configuration files using your preferred text editor. This way of doing -can bring much power to advanced users, but requires an access to the -server's files. It also requires that the user double-check his -modifications, or use the web interface to ensure the validity and -correctness of his formats. - -In this manual we will show both ways of doing. For each explication we -show first how to do it through the web interface, then how to do it by -manipulating the configuration files. Non-power users can stop reading -as soon as they encounter the text "For developers and adventurers -only". - -We generally recommend to use the web interface, excepted for writing -format elements. - -2. Output Formats ------------------ - -As you potentially have a huge amount of bibliographic records, you -cannot specify manually for each of them how it should be formatted. -This is why you can define rules that will allow BibFormat to understand -which kind of formatting to apply to a given record. You define this set -of rules in what is called an "output format". - -You can have different output formats, each with its own -characteristics. For example you certainly want that when multiple -bibliographic records are displayed at the same time (as it happens in -search results), only short versions are shown to the user , while a -detailed record is preferable when a single record is displayed, -whatever the type of the record. - -You might also want to let your users decide which kind of output -they want. For example you might need to display HTML for regular web -browsing, but would also give a BibTeX version of the bibliographic -reference for direct inclusion in a LaTeX document. - -To summarize, an output format groups similar kind of formats, -specifying which kind of formatting has to be done, but not how it has -to be done. - -2.1 Add an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~ - -To add a new output format, go to the `Manage Output -Formats `__ -page and click on the "Add New Output Format" button at the bottom of -the page. The format has been created. You can then specify the -attributes of the output format. See `Edit the Attributes of an Output -Format <#attrsOutputFormat>`__ to learn more about it. - -**For developers and adventurers only:** - -Alternatively you can directly add a new output format file into the -/etc/bibformat/outputs/ directory of your Invenio installation, if you -have access to the server's files. Use the format extension .bfo for -your file. - -You should also check that user ``www-data`` has read/write access to -the file, if you want to be able to modify the rules through the web -interface. - -2.2 Remove an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To remove an output format, go to the `Manage Output -Formats `__ -page and click on the "Delete" button facing the output format you want -to delete. If you cannot click on the button (the button is not -enabled), this means that you do not have sufficent priviledge to do so -(Format is protected. Contact the administrator of the system). - -**For developers and adventurers only:** - -You can directly remove an output format from the -/etc/bibformat/outputs/ directory of your Invenio installation. However -you must make sure that it is removed from the tables ``format`` and -``formatname`` in the database, so that other modules know that it is -not longer available. - -2.3 Edit the Rules of an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When you create a new output format, you can at first only specify the -default template, that is the one which is used when all rules fail. In -the case of a basic output format, this is enough. You can however add -other rules, by clicking on the "Add New Rule" button. - -Once you have added a rule, you can fill it with a condition, and a -template that should be used if the condition is true. For example the -rule - -|Rule: Use template Picture HTML Detailed if field 980__a is equal to PICTURE| - -will use template named "Picture HTML Detailed" if the field -``980__a`` of the record to format is equal to "Picture". Note that text -"PICTURE" will match any letter case like "picture" or "Picture". -Leading and trailing spaces are ignored too (" Picture " will match -"PICTURE"). - -**Tips:** you can use a regular expression as text. For example -"PICT.\*" will match "pictures" and "PICTURE". - -|Reorder rules using arrows| - -The above configuration will use format template "Default HTML Detailed" -if all above rules fail (in that case if field 980\_\_a is different -from "PICTURE"). If you have more rules, you decide in which order the -conditions are evaluated. You can reorder rules by clicking on the small -arrows on the left of the rules. - -Note that when you are migrating your output formats from the old PHP -BibFormat, you might not have translated all the formats to which your -output formats refers. In that case you should use -``defined in old BibFormat`` option in the format templates menu, to -make BibFormat understand that a match for this rule must trigger a call -to the *Behaviour* of the old BibFormat. See section on `Run old and new -formats side by side <#runSideBySide>`__ for more details on this. - -**For developers and adventurers only:** - -To write an output format, use the following syntax: - -First you define which field code you put as the conditon for the -rule. You suffix it with a column. Then on next lines, define the values -of the condition, followed by --- and then the filename of the template -to use: - -:: - - tag 980__a: - PICTURE --- PICTURE_HTML_BRIEF.bft - PREPRINT --- PREPRINT_HTML_BRIEF.bft - PUBLICATION --- PUBLICATION_HTML_BRIEF.bft - -This means that if value of field 980\_\_a is equal to PICTURE, then we -will use format template PICTURE\_HTML\_BRIEF.bft. Note that you must -use the filename of the template, not the name. Also note that spaces at -the end or beginning are not considered. On the following lines, you can -either put other conditions on tag 980\_\_a, or add another tag on which -you want to put conditions. - -At the end you can add a default condition: - -:: - - default: PREPRINT_HTML_BRIEF.bft - -which means that if no condition is matched, a format suitable for -Preprints will be used to format the current record. - -The output format file could then look like this: - -:: - - tag 980__a: - PICTURE --- PICTURE_HTML_BRIEF.bft - PREPRINT --- PREPRINT_HTML_BRIEF.bft - PUBLICATION --- PUBLICATION_HTML_BRIEF.bft - - tag 8560_f: - .*@cern.ch --- SPECIAL_MEMBER_FORMATTING.bft - - default: PREPRINT_HTML_BRIEF.bft - -You can add as many rules as you want. Keep in mind that they are read -in the order they are defined, and that only first rule that matches -will be used. Notice the condition on tag 8560\_f: it uses a regular -expression to match any email address that ends with @cern.ch (the -regular expression must be understandable by Python) - -2.4 Edit the Attributes of an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An output format has the following attributes: - -- ``code``: a short identifier that is used to identify the output - format. It must be unique and contain a maximum of 6 letters. Note - that the **code is not case sensitive** ("HB" is equal to "hb"). -- ``content type``: this is the content type of the format, specified - in Mime. For example if you were to produce an Excel output, you - could use ``application/ms-excel`` as content type. If a content type - is specified, Invenio will not print the usual header and footerfor - the page, but will trigger a download in the client's browser when - viewing the page (Unless the browser handles this content type). -- ``name``: a generic name to display in the interface for this output - format. -- (\*) ``name``: internationalized names for the output format, used - for displaying localized name in the search interface. -- ``description``: an optional description for the output format. - -**Please read this information regarding output format codes:** There -are some reserved codes that you should not use, or at least be aware of -when choosing a code for your output format. The table below summarizes -these special words: - -+--------------------------------------+--------------------------------------+ -| Code Purpose | | -+======================================+======================================+ -| HB | HD | -| Used for displaying list of results | Used when no format is specified | -| of a search. | when viewing a record. | -+--------------------------------------+--------------------------------------+ - -**For developers and adventurers only:** - -Excepted for the code, output format attributes cannot be changed in the -output format file. These attributes are saved in the database. As for -the ``code``, it is the name of the output format file, without its -``.bfo`` extension. If you change this name, do not forget to propagate -the modification in the database. - -2.5 Check the Dependencies an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To check the dependencies of an output format on format templates, -format elements and tags, go to the `Manage Output -Formats `__ -page, click on the output format you want to check, and then in the menu -click on "Check Dependencies". - -|Check Dependencies menu| - -The next page shows you: - -- the format templates which might be called by the rules of the output format -- the elements used in each of these templates -- the Marc tags involved in these elements - -Note that some Marc tags might be omitted. - -2.6 Check the Validity an Output Format -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To check the validity of an output format, simply go to the `Manage Output -Formats `__ -page, and look at the column 'status' for the output format you want to -check. If message "Ok" is there, then no problem was found with the -output format. If message 'Not Ok' is in the column, click on it to see -the problems that have been found for the output format. - -3. Format Templates -------------------- - -A format template defines how a record should be formatted. For example -it specifies which fields of the record are to be displayed, in which -order and with which visual attributes. Basically the format template is -written in HTML, so that it is easy for anyone to edit it. BibFormat -also has support for XSLT for formatting. Read more `about XSL format -templates here <#xslFormatTemplate>`__. - -3.1 Add a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~ - -To add a new format template, go to the `Manage Format -Templates `__ -page and click on the "Add New Format Template" button at the bottom of -the page. The format has been created. You can then specify the -attributes of the format template, or ask to make a copy of an existing -format. See `Edit the Attributes of a Format -Template <#attrsFormatTemplate>`__ to learn more about editing the -attributes. - -**For developers and adventurers only:** - -Alternatively you can directly add a new format template file into the -/etc/bibformat/format\_templates/ directory of your Invenio -installation, if you have access to the server's files. Use the format -extension .bft for your file. - -You should also check that user ``www-data`` has read/write access to -the file, if you want to be able to modify the code and the attributes -of the template through the web interface. - -3.2 Remove a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To remove a format template, go to the `Manage Format -Templates `__ -page and click on the "Delete" button facing the format template you -want to delete. If you cannot click on the button (the button is not -enabled), this means that you do not have sufficent priviledge to do so -(Format is protected. Contact the administrator of the system). - -**For developers and adventurers only:** - -You can directly remove the format template from the -/etc/bibformat/format\_templates/ directory of your Invenio -installation. - -3.3 Edit the Code of a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can change the formatting of records by modifying the code of a -template. - -To edit the code of a format template go to the `Manage Format -Templates `__ -page. Click on the format template you want to edit to load the template -editor. - -The format template editor contains three panels. The left upper panel -is the code editor. This is were you write the code that specifies the -formatting of a template. The right-most panel is a short documentation -on the "bricks" you can use in your format template code. The panel at -the bottom of the page allows you to preview the template. - -|Template Editor Page| - -The following sections explain how to write the code that specifies the -formatting. - -3.4 Basic Editing -^^^^^^^^^^^^^^^^^ - -The first thing you have to know before editing the code is that -everything you write in the code editor is printed as such by BibFormat. -Well almost everything (as you will discover later). - -For example if you write "My Text", then for every record the output -will be "My Text". Now let's say you write "My Text": the output -will still be "My Text", but as we display in a web browser, it -will look like "**My Text**\ " (The browser interprets the text inside -tags as "bold". Also note that the look may depend on the CSS -style of your page). - -Basically it means that you can write HTML to do the formatting. If you -are not experienced with HTML you can use an HTML editor to create your -layout, and the copy-paste the HTML code inside the template. - -Do not forget to save your work by clicking on the save button before -you leave the editor! - -**For developers and adventurers only:** - -You can edit the code of a template using exactly the same syntax as in -the web interface. The code of the template is in the template file -located in the /etc/bibformat/format\_templates/ directory of your -Invenio installation. You just have to take care of the attributes of -the template, which are saved in the same file as the code. See `Edit -the Attributes of a Format Template <#attrsFormatTemplate>`__ to learn -more about it. - -3.5 Use Format Elements -^^^^^^^^^^^^^^^^^^^^^^^ - -To add a dynamic behaviour to your format templates, that is display for -example a different title for each record or a different background -color depending on the type of record, you can use the format elements. - -Format elements are the smart bricks you can copy-paste in your code to -get the attributes of template that change depending on the record. A -format element looks like a regular HTML tag. - -For example, to print the title of a record, you can write -```` in your template code where you want to diplay the -title - -Format elements can take values as parameters. This allows to customize -the behaviour of an element. For example you can write -````, and BibFormat will take care of -printing the title for you, with prefix "Title: ". The difference -between ``Title: `` and ```` -is that the first option will always write "Title: " while the second -one will only print "Title: " if there exist a title for the record in -the database. Of course there are chances that there is always a title -for each record, but this can be useful for less common fields. - -Some parameters are available for all elements. This is the case for the -following ones: - -- ``prefix``: a prefix printed only if the record has a value for the - element. -- ``suffix``: a suffix printed only if the record has a value for the - element. -- ``default``: a default value printed if the record has no value for - the element. In that case ``prefix`` and ``suffix`` are not printed. - -Some parameters are specific to elements. To get information on all -available format elements you can read the `Format Elements -Documentation `__, -which is generated dynamically for all existing elements. it will show -you what the element do and what parameters it can take. - -While format elements looks like HTML tags, they differ in the -followings ways from traditional ones: - -- A format element is a single tag: you cannot have - ``some text`` but only ````. -- The values of the parameters accept any characters, including < and - >. The only limitation is that you cannot use the type of quotes that - delimit that value: you can have for example - ```` or - ````, but not - ````. -- Format elements names always start with ``BFE_``. -- Format element can expand on multiple lines. - -**Tips:** you can use the special element ```` to -print the value of any field of a record in your templates. This -practice is however not recommended because it would necessitate to -revise all format templates if you did change the meaning of the MARC -code schema. - -3.6 Preview a Format Template -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To preview a format template go to the `Manage Format -Templates `__ -page and click on the format template you want to preview to open the -template editor. The editor contains a preview panel at the bottom of -the page. - -|Preview Panel| - -Simply click on " Reload Preview" button to preview the template (you -do not need to save the code before previewing). - -Use the "Language" menu to preview the template in a given language - -You can fill in the "Search Pattern" field to preview a specific record. -The search pattern uses exactly the same syntax as the one used in the -web interface. The only difference with the regular search engine is -that only the first matching record is shown. - -**For developers and adventurers only:** - -If you do not want to use the web interface to edit the templates but -still would like to get previews, you can open the preview frame of any -format in a new window/tab. In this mode you get a preview of the -template (if it is placed in the /etc/bibformat/format\_templates/ -directory of your Invenio installation). The parameters of the preview -are specified in the url: - -- ``bft``: the filename of the format template to preview -- ``ln``: the language to use for the preview -- ``pattern_for_preview``: the search pattern to use for the preview - -3.7 Internationalization (i18n) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can add translations to your format templates. To do so enclose the -text you want to localize with tags corresponding to the two letters of -the language. For example if we want to localize "title", write -``Title``. Repeat this for each language in which you want to -make "title" available: ``TitleTitreTitel``. -Finally enclose everything with `` `` tags: -``TitleTitreTitel`` - -For each group only the text in the user's language is displayed. -If user's language is not available in the group, your default -Invenio language is used. - -3.8 Escaping special HTML/XML characters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -By default, BibFormat escapes all values returned by format elements. As -a format template designer, you can assume in almost all cases that the -values you get from a format element will be escaped for you. For -special cases, you can set the parameter ``escape`` of the element to -'0' when calling it, to make BibFormat understand that it must not -escape the values of the element, or to '1' to force the escaping. - -See the `complete list of escaping modes <#listofescapingmodes>`__. - -For example ```` will return: - -:: - - [...]We find that for spatially-flat cosmologies, background lensing - clusters with reasonable mass-to-light ratios lying in the - redshift range 0<1 are strongly excluded, [...] - -while ```` will return: - -:: - - [...]We find that for spatially-flat cosmologies, background lensing - clusters with reasonable mass-to-light ratios lying in the - redshift range 0<1 are strongly excluded, [...] - -In most cases, you will not set ``escape`` to 1, nor 0, but just let the -developer of the element take care of that for you. - -Please note that values given in special parameters ``prefix``, -``suffix``, ``default`` and ``nbMax`` are never escaped, whatever the -value of ``escape`` is (but other parameters will). You have to take -care of that in your format template, as well as of all other values -that are not returned by the format elements. - -3.9 Edit the Attributes of a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To edit the attributes of a format template go to the `Manage Format -Templates `__ -page, click on the format template you want to edit, and then in the -menu click on "Modify Template Attributes". - -A format template contains two attributes: - -- ``Name``: the name of the template -- ``Description``: a short description of the template - -Note that changing these parameters has no impact on the formatting. -Their purpose in only to document the template. - -If the name you have chosen already exists for another template, you -name will be suffixed with an integer so that the name is unique. - -You should also be aware that if you change the name of a format -template, all output formats that were linking to this template will be -changed to match the new name. - -**For developers and adventurers only:** - -You can change the attributes of a template by editing its file in the -/etc/bibformat/format\_templates/ directory of your Invenio -installation. The attributes must be enclosed with tags -`` `` and `` `` and should -ideally be placed at the beginning of the file. - -Also note that the admin web interface tries to keep the name of the -template in sync with the filename of the template. If the name is -changed through the web interface, the filename of the template is -changed, and all output formats that use this template are updated. You -have to do update output formats manually if you change the filename of -the template without the web interface. - -3.10 Check the Dependencies of a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To check the dependencies of a format template go to the `Manage Format -Template `__ -page, click on the format template you want to check, and then in the -menu click on "Check Dependencies". - -|Check Dependencies menu| - -The next page shows you: - -- The output formats that use this format template -- the elements used in the template (and Marc tags use in these - elements in parentheses) -- A summary of all the Marc tags involved in the elements of the - template - -Note that some Marc tags might be omitted. - -3.11 Check the Validity a Format Template -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To check the validity of a format template, simply go to the `Manage -Format -Templates `__ -page, and look at the column 'status' for the format template you want -to check. If message "Ok" is there, then no problem was found with the -template. If message 'Not Ok' is in the column, click on it to see the -problems that have been found for the template. - -3.12 XSL Format Templates -~~~~~~~~~~~~~~~~~~~~~~~~~ - -In addition to the HTML-like syntax introduced in previous sections, -BibFormat also has support for server-side XSL transformation. Although -you can do all the formatting using this custom HTML syntax, there are -cases where an XSL stylesheet might be preferred. XSLT is for example a -natural choice when you need to output complex XML, especially when your -XML has a deep tree structure. You might also prefer using XSLT if you -already feel comfortable with XSL syntax. - -XSL format templates are written using regular XSL. The template file -has to be placed in the same folder as regular format template files, -and its file extension must be ``.xsl``. The XSL template are also -visible through the web interface, as any regular format template file. -However, some functions like the "Dependencies checker" or the -possibility to create a template or edit its attributes are not -available for the XSL templates. - -In BibFormat XSL you have access to the following functions, provided -you have declared ``xmlns:fn="http://cdsweb.cern.ch/bibformat/fn"`` in -your stylesheet: - -**``fn:modification_date(recID)``** - Returns the record modification date. Eg: - ```` returns - modification date of record 445 - -**``fn:creation_date(recID)``** - Returns the record creation date. Eg: - ```` returns creation - date of record 445 - -**``fn:eval_bibformat(recID, bibformat_template_code)``** - Returns the results of the evaluation of the format template code. - Eg: - ```` - returns the url of the current record. The parameter - ``bibformat_template_code`` is regular code used inside BibFormat - format templates, with ``<`` escaped as ``<`` and ``"``\ (quotes) - escaped as ``"`` - -Finally, please note that you will need to install a supported XSLT -parser in order to format using XSL stylesheets. - -4. Format Elements ------------------- - -Format elements are the bricks used in format templates to provide -dynamic content to the formatting process. Their purpose is to allow non -computer literate persons to easily integrate data from the records in -the database into their templates. - -Format elements are typically written in Python (there is an exception -to that point which is dicussed in `Add a Format -Element <#addFormatElement>`__). This brings great flexibily and power -to the formatting process. This however restricts the creation of format -elements to developers. - -4.1 Add a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~ - -The most typical way of adding a format element is to drop a ``.py`` -file in the lib/python/invenio/bibformat\_elements directory of your -Invenio installation. See `Edit the Code of a Format -Element <#codeFormatElement>`__ to learn how to implement an element. - -The most simple way to add a format element is to add a en entry in the -"`Logical -Fields `__\ " -management interface of the BibIndex module. When BibFormat cannot find -the Python format element corresponding to a given name, it looks into -this table for the name and prints the value of the field declared for -this name. This lightweight way of doing is straightforward but does not -allow complex handling of the data (it limits to printing the value of -the field, or the values of the fields if multiple fields are declared -under the same label). - -4.2 Remove a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To remove a Python format element simply remove the corresponding file -from the lib/python/invenio/bibformat\_elements directory of your -Invenio installation. - -To remove a format element declared in the "`Logical -Fields `__\ " -management interface of the BibIndex module simply remove the entry from -the table. - -4.3 Edit the Code of a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section only applies to Python format elements. Basic format -elements declared in "`Logical -Fields `__\ " -have non configurable behaviour. - -A format element file is like any regular Python program. It has to -implement a ``format_element`` function, which returns a ``string`` and -takes at least ``bfo`` as first parameter (but can take as many others -as needed). - -Here is for example the code of the "bfe\_title.py" element: - -:: - - def format_element(bfo, separator=" "): - """ - Prints the title of a record. - - @param separator separator between the different titles - """ - titles = [] - - title = bfo.field('245__a') - title_remainder = bfo.field('245__b') - - titles.append( title + title_remainder ) - - title = bfo.field('246__a') - if len(title) > 0: - titles.append( title ) - - title = bfo.field('246_1a') - if len(title) > 0: - titles.append( title ) - - return separator.join(titles) - -In format templates this element can be called like a function, using -HTML syntax: ```` - -Notice that the call uses (almost) the filename of your element. To -find out which element to use, BibFormat tries different filenames until -the element is found: it tries to - -#. ignore the letter case -#. replace underscore with spaces -#. remove the BFE\_ from the name - -This means that even if the filename of your element is "my element.py", -BibFormat can resolve the call in a format -template. This also means that you must take care no to have two format -elements filenames that only differ in term of the above parameters. - -The ``string`` returned by the ``format_element`` function corresponds -to the value that is printed instead of the format element name in the -format template. - -The ``bfo`` object taken as parameter by ``format_element`` function -stands for BibFormatObject: it is an object that represents the context -in which the formatting takes place. For example it allows to retrieve -the value of a given field for the record that is being formatted, or -the language of the user. We see the details of the BibFormatObject -further below. - -The ``format_element`` function of an element can take other parameters, -as well as default values for these parameters. The idea is that these -parameters are accessible from the format template when calling the -elements, and allow to parametrize the behaviour of the format element. - -It is very important to document your element: this allows to generate a -documentation for the elements accessible to people writing format -templates. It is the only way for them to know what your element do. The -key points are: - -- Provide a docstring for the ``format_element`` function -- For each of the parameters of the ``format_element`` function (except - for ``bfo``) as provide a description using a Java-like doc syntax in - the doc string: ``@param my_param: description for my param`` (one line per - parameter) -- You can use one ``@see`` followed by a comma separated list of - elements filenames to provide a reference to other elements of - interests related to this one: ``@see my_element1.py, my element2.py`` - -Typically you will need to get access to some fields of a record to -display as output. There are two ways to this: you can access the -``bfo`` object given as parameter and use the provided (basic) -accessors, or import a dedicated module and use its advanced -functionalities. - -**Method 1: Use accessors of ``bfo``**: - -``bfo`` is an instance of the ``BibFormatObject`` class. The -following methods are available: - -- ``get_record()``: Returns the record of this BibFormatObject instance - as a BibRecord structure. Allows advanced access on the structure - using ``BibRecord``. -- ``control_field(tag)``: Returns the value of control field given by - MARC ``tag``. -- ``field(tag)``:Returns the value of the field corresponding to MARC - ``tag``. If the value does not exist, return empty string. -- ``fields(tag)``: Returns the list of values corresonding to MARC - ``tag``.If tag has an undefined subcode (such as 999C5), the function - returns a list of dictionaries, whoose keys are the subcodes and the - values are the values of tag.subcode. If the tag has a subcode, - simply returns list of values corresponding to tag. -- ``kb(kb, string, default="")``: Returns the value of the ``string`` - in the knowledge base ``kb``. If kb does not exist or string does not - exist in kb, returns ``default`` string. - -You can also get access to other information through ``bfo``, such as -the language in which the formatting should occur with ``bfo.lang``. To -learn more about the possibilities offered by the ``bfo``, read the -`BibFormat APIs `__ - -**Method 2: Use module ``BibRecord``**: - -BibRecord is a module that provides advanced functionalities -regarding access to the field of a record ``bfo.get_record()`` returns a -structure that can be understood by BibRecord's functions. Therefore you -can import the module's functions to get access to the fields you want. - -4.4 Preview a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can play with a format element parameters and see the result of the -element directly in the `format elements -documentation `__: -for each element, under the section "See also", click on "Test this -element". You are redirected to a page where you can enter a value for -the parameters. A description is associated with each parameter as well -as an indication of the default value of the parameter if you do not -provide a custom value. Click on the "Test!" button to see the result of -the element with your parameters. - -4.5 Internationalization (i18n) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can follow the standard internationalization procedure in use -accross Invenio sources. For example the following code will get you the -translation for "Welcome" (assuming "Welcome" has been translated): - -:: - - from invenio.base.i18n import gettext_set_language - - ln = bfo.ln - _ = gettext_set_language(ln) - - translated_welcome = _("Welcome") - -Notice the access to ``bfo.ln`` to get access to the current language of -the user. For simpler translations or behaviour depending on the -language you can simply check the value ``bfo.ln`` to return your custom -text. - -4.6 Escaping special HTML/XML characters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In most cases, that is cases where your element does not return HTML -output, you do not have to take any particular action in order to escape -values that you output: the BibFormat engine will take care of escaping -the returned value of the element for you. In cases where you want to -return text that should not be escaped (for example when you return HTML -links), you can make the formatting engine know that it should not -escape your value. This is done by implementing the -``escape_values(bfo)`` function in your element, that will return (int) -0 when escape should not be done (or 1 when escaping should be done): - -:: - - def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - -Note that the function is given a ``bfo`` object as parameter, such -that you can do additional testing if your element should really return -1 or 0 (for very special cases). - -Also note that the behavior defined by the ``escape_values()`` -function will be overriden by the ``escape`` parameter used in the -format template if it is specified. - -Finally, be cautious when you disable escaping: you will have to take -care of escaping values "manually" in your format element code, in order -to avoid non valid outputs or XSS vulnerabilities. This can be done -easily when using the ``field``, ``fields`` and ``controlfield`` -functions of bfo with ``escape`` parameter: - -:: - - title = bfo.field('245__a', escape="1") - abstract = bfo.field('520__a', escape="2") - -The ``escape`` parameter can be one of the following values: - -- 0 - no escaping -- 1 - escape all HTML characters (escaped chars are shown as escaped) -- 2 - remove unsafe HTML tags to avoid XSS, but keep basic one (such as -
) This is particularly useful if you want to store HTML text in - your metadata but still want to escape some tags to prevent XSS - vulnerabilities. Note that this method is slower than basic escaping - of mode 1. - Escaped tags are removed. -- 3 - mix of mode 1 and mode 2. If field\_value starts with - , then use mode 2. Else use mode 1. -- 4 - remove all HTML/XML tags -- 5 - same as 2, but allows more tags, like -- 6 - same as 3, but allows more tags, like -- 7 - mix of mode 0 and mode 1. If field\_value starts with - , then use mode mode 0. Else use mode 1. -- 8 - same as mode 1, but also escape double-quotes -- 9 - same as mode 4, but also escape double-quotes - -These modes are the same for ``escape_values(bfo)`` function. - -You can also decide not to use the ``escape`` parameter and escape -values using any other Python function/library you want to use (such as -``cgi.escape()``). - -As a BibFormat element developer you can also override the default -``escape`` parameter of your format elements: that is especially useful -if you want to provide a way for format templates editors to call your -element with a custom escaping mode that should not escape the whole -output of your element. The ``bfe_abstract.py`` element is an example of -code that overrides the ``escape`` parameter. - -4.7 Edit the Attributes of a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A format element has mainly four kinds of attributes: - -- Name: it corresponds to the filename of the element. -- Description: the description is in the ``docstring`` of the - ``format_element`` function (excepted lines prefixed with ``@param`` - and ``@see``). -- Parameters descriptions: for each parameter of the ``format_element`` - function, a line beginning with ``@param`` *parameter\_name* and - followed by the description of the parameter is present in the - ``docstring`` of the ``format_element`` function. -- Reference to other elements: one line beginning with ``@see`` and - followed by a list of comma-separated format elements filenames in - the in the ``docstring`` of the ``format_element`` function provides - a link to related elements. - -4.8 Check the Dependencies of a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two ways to check the dependencies of a format element. The -simplest way is to go to the `format elements -documentation `__ -and click on "Dependencies of this element" for the element you want to -check. - -The second method to check the dependencies of an element is through -regular unix tools: for example -``$ grep -r -i 'bfe_your_element_name' .`` inside the format templates -directory will tell you which templates call your element. - -4.9 Check the Validity of a Format Element -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two ways to check the validity of an element. The simplest one -is to go to the `format elements -documentation `__ -and click on "Correctness of this element" for the element you want to -check. - -The second method to check the validity of an element is through regular -Python methods: you can for example import the element in the -interactive interpreter and feed it with test parameters. Notice that -you will need to build a BibFormatObject instance to pass as ``bfo`` -parameter to the ``format_element`` function of your element. - -4.10 Browse the Format Elements Documentation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Go to the `format elements -documentation `__. -There is a summary of all available format elements at the top of the -page. You can click on an element to go to its detailed description in -the second part of the page. - -Each detailed documentation shows you: - -- A description of what the element does. -- A list of all parameters you can use for this element. -- For each parameter, a description and the default value when - parameter is ommitted. -- A link to a tool to track the dependencies of your element. -- A link to a tool to check the correctness of your element. -- A link to a tool to test your element with custom parameters. - -5. Run BibReformat ------------------- - -While records can be formatted on-the-fly using BibFormat, it is usually -necessary to preformat the records in order to decrease the load of your -server. To do so, use the ``bibreformat`` command line tool. - -5.1 Run BibReformat -~~~~~~~~~~~~~~~~~~~ - -The following options are available for running ``bibreformat``: - -:: - - Usage: bibreformat [options] - -u, --user=USER User name to submit the task as, password needed. - -h, --help Print this help. - -V, --version Print version information. - -v, --verbose=LEVEL Verbose level (0=min,1=normal,9=max). - -s, --sleeptime=SLEEP Time after which to repeat tasks (no) - -t, --time=DATE Moment for the task to be active (now). - -a, --all All records - -c, --collection Select records by collection - -f, --field Select records by field. - -p, --pattern Select records by pattern. - -o, --format Specify output format to be (re-)created. (default HB) - -n, --noprocess Count records to be processed only (no processing done) - Example: bibreformat -n Show how many records are to be bibreformated. - -For example, to reformat all records in HB (=HTML brief) format, you'd -launch: - -:: - - $ bibreformat -a -oHB - -and you watch the progress of the process via ``bibsched``. - -Note that BibReformat understands ``-p``, ``-f``, and ``-c`` arguments -that enable you to easily reformat only the records you need. For -example, to reformat the Pictures collection, launch: - -:: - - $ bibreformat -cPictures -oHB - -or to reformat HD (=HTML detailed) format for records #10 to #20, you -launch: - -:: - - $ bibreformat -p"recid:10->20" -oHD - -Last but not least, if you launch bibreformat without arguments: - -:: - - $ bibreformat - -it will process all the records that have been modified since the last -run of BibReformat, as well as all newly inputted records. This is -suitable for running BibReformat in a periodical daemon mode via -BibSched. See our `HOWTO Run Your Invenio -Installation `__ guide for -more information. - -6. Appendix ------------ - -6.1 MARC Notation in Formats -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The notation for accessing fields of a record are quite flexible. You -can use a syntax strict regarding MARC 21, but also a shortcut syntax, -or a syntax that can have a special meaning. - -The MARC syntax is the following one: -``tag[indicator1][indicator2] [$ subfield]`` where ``tag`` is 3 digits, -``indicator1`` and ``indicator2`` are 1 character each, and ``subfield`` -is 1 letter. - -For example to get access to an abstract you can use the MARC notation -``520 $a``. You can use this syntax in BibFormat. However you can also: - -- Omit any whitespace character (or use as many as you want) -- Omit the ``$`` character (or use as many as you want) -- Omit or use both indicators. You cannot specify only one indicator. - If you need to use only one, use underscore ``_`` character for the - other indicator. -- Use percent '``%``\ ' instead of any character to specify all ("don't - care" or wildcard character) for that character. - -6.2 Migrating from Previous BibFormat -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The new Python BibFormat formats are not backward compatible with the -previous formats. New concepts and capabilities have been introduced and -some have been dropped. If you have not modified the "Formats" or -modified only a little bit the "Behaviours" (or modified "Knowledge -Bases"), then the transition will be painless and automatic. Otherwise -you will have to manually rewrite some of the formats. This should -however not be a big problem. - -The first thing you should do is to read the `Five Minutes Introduction -to BibFormat <#shortIntro>`__ to understand how the new BibFormat works. -We also assume that you are familiar with the concepts of the old -BibFormat. As the new formats separate the presentation from the -business logic (i.e. the bindings to the database), it is not possible -to automatically handle the translation. This is why you should at least -be able to read and understand the formats that you want to migrate. - -Differences between old and new BibFormat -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The most noticeable differences are: - - a) "Behaviours" have been renamed "Output formats". - b) "Formats" have been renamed "Format templates". They are now written in HTML. - c) "User defined functions" have been dropped. - d) "Extraction rules" have been dropped. - e) "Link rules" have been dropped. - f) "File formats" have been dropped. - g) "Format elements" have been introduced. They are written in - Python, and can simulate c), d) and e). - h) Formats can be managed through web interface or through - human-readable config files. - i) Introduction of tools like validator and dependencies checker. - j) Better support for multi-language formatting. - -Some of the advantages are: - + Management of formats is much clearer and easier (less concepts, more tools). - + Writing formats is easier to learn : less concepts to learn, - redesigned work-flow, use of existing well known and well documented - languages. - + Editing formats is easier: You can use your preferred HTML editor - such as Emacs, Dreamweaver or Frontpage to modify templates, or any text - editor for output formats and format elements. You can also use the - simplified web administration interface. - + Faster and more powerful templating system. - + Separation of business logic (output formats, format elements) and - presentation layer (format templates). This makes the management of - formats simpler. - -The disadvantages are: - - No backward compatibility with old formats. - - Stricter separation of business logic and presentation layer: - no more use of statements such as if(), forall() inside templates, - and this requires more work to put logic inside format elements. - -Migrating *behaviours* to *output formats* -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Behaviours were previously stored in the database and did require to use -the evaluation language to provide the logic that choose which format to -use for a record. They also let you enrich records with some custom -data. Now their use has been simplified and rectricted to equivalence -tests on the value of a field of the record to define the format -template to use. - -For example, the following behaviour: - -**CONDITIONS** - -**0** - -**$980.a="PICTURE"** - -**Action (0)** - -| " -|  " $001 " -|    -|  hb  -|  "  -| xml\_text(format("PICTURE\_HTML\_BRIEF")) -| "   -|   -| " - -  - -**100** - -**""=""** - -**Action (0)** - -| " -|  " $001 " -|    -|  hb  -|  "  -| xml\_text(format("DEFAULT\_HTML\_BRIEF")) -| "   -|   -| " - -  - -translates to the following output format (in textual configuration -file): - -`` tag 980__a: PICTURE --- Picture_HTML_brief.bft default: Default_HTML_brief.bft`` - -| or visual representation through web interface: -| |Image representation of HB output format| - -Migrating *formats* to *format templates* and *format elements* -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The migration of formats is the most difficult part of the migration. -You will need to separate the presentation code (HTML) from the business -code (iterations, tests and calls to the database). Here are some tips -on how you can do this: - -- If you want to save the time of unescaping all HTML characters and - understanding how the layout should look like, just go with your web - browser to a formatted version of the format in your Invenio - installation, and copy the source of the web page. Identify the parts - of the HTML code which are specific to the current record, and - replace them with a call to the corresponding format element. -- If you have made small modifications to the old default provided - formats, we suggest that you use the new provided ones and modify - them according to your needs. - -Migrating *UDFs* and *Link rules* -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*User Defined Functions* and *Link rules* have been dropped in the new -BibFormat. These concepts have no reasons to be as they can be fully -implemented in the *format elements*. For example the ``AUTHOR_SEARCH`` -link rule can directly be implemented in the ``Authors.bfe`` element. - -As for the UDFs, most of them are directly built-in functions of Python. -Whenever a special function as to be implemented, it can be defined in a -regular Python file and used in any element. - -The Migration Kit -^^^^^^^^^^^^^^^^^ - -The migration kit is only available in older versions of Invenio. - -To enable the migration kit in this release, you need to copy the -required files from an older release. - -Run old and new formats side by side -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This was possible by default only in older versions of Invenio. - -To enable this functionality in this release, you need to copy the -required files from an older release. - -6.3 Integrating BibFormat into Dreamweaver MX -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -BibFormat templates have been thought to be editable in custom HTML -editors. We propose in this section a way to extend one particular -editor, Dreamweaver. - -Make Dreamweaver Recognize Format Elements in Layout View -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To make Dreamweaver understand the format elements and display an icon -for each of them in the layout editor, you must edit a Dreamweaver -configuration file named ``Tags.xml`` located inside -/Configuration/ThirdPartyTags directory of your Dreamweaver installation -folder. At the end of this file, copy-paste the following lines: - -:: - - - - - -Also copy this icon |bibformat.gif| in the same directory as -``Tags.xml`` (right-click on icon, or ctrl-click on one-button mouse, -and "Save Image As..."). Make sure the downloaded image is named -"``bibformat.gif``\ ". - -Note that Dreamweaver might not recognize Format Elements when complex -formatting is involved due to these elements. - -Add a Format Elements Floating Panel -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can add a floating panel that will you to insert Format Elements in -your document and read the documentation of all available Format -Elements. - -The first step is to declare in which menu of Dreamweaver this floating -panel is going to be available. To do so, edit file "``Menu.xml``\ " -located inside /Configuration/Menus of your Dreamweaver application -directory and copy-paste the following line in the menu you want -(typically inside tag ``'menu'`` with attribute -``id='DWMenu_Window_Others')``: - -:: - - - -Once this is done, you can `download the floating -palette `__ -(if file opens in your browser instead of downloading, right-click on -icon, or ctrl-click on one-button mouse, and "Save Target As...") and -move the dowloaded file "``BibFormat_floater.html``\ " (do not rename -it) into /Configuration/Floaters directory of your Dreamweaver -application folder. - -To use the BibFormat floating panel, open Dreamweaver, and choose -``Window > Others > BibFormat Elements``. - -Whenever a new version of the palette is available, you can skip the -edition of file "``Menu.xml``\ " and just replace the old -"``BibFormat_floater``\ " file with the new one. - -6.4 FAQ -~~~~~~~ - -Why do we need output formats? Wouldn't format templates be sufficient? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -As you potentially have a lot of records, it is not conceivable to -specify for each of them which format template they should use. This is -why this rule-based decision layer has been introduced. - -How can I protect a format? -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -As a web user, you cannot protect a format. If you are administrator of -the system and have access to the format files, you can simply use the -permission rights of your system, as BibFormat is aware of it. - -Why cannot I edit/delete a format? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The format file has certainly been protected by the administrator of the -server. You must ask the administrator to unprotect the file if you want -to edit it. - -How can I add a format element from the web interface? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Format elements cannot be added, removed or edited through the web -interface. This limitation has been introduced to limit the security -risks caused by the upload of Pythonic files on the server. The only -possibility to add a basic format element from the web interface is to -add a en entry in the "`Logical -Fields `__\ " -management interface of the BibIndex module (see `Add a Format -Element <#addFormatElement>`__) - -Why are some Marc codes omitted in the "Check Dependencies" pages? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When you check the dependencies of a format, the page reminds you that -some use of Marc codes might not be indicated. This is because it is not -possible (or at least not trivial) to guess that the call to -``field(str(5+4)+"80"+"__a")`` is equal to a call to -``field("980__a")``. You should then not completely rely on this -indication. - -How are displayed deleted record? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -By default, Invenio displays a standard "The record has been deleted." -message for all output formats with a 'text/html' content type. Your -output format, format templates and format elements are bypassed by the -engine. However, for more advanced output formats, Invenio goes through -the regular formatting process and let your formats do the job. This -allows you to customize how a record should be displayed once it has -been deleted. - -Why are some format elements omitted in the "Knowledge Base Dependencies" page? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When you check the dependencies of a knowledge base, the page reminds -you that format elements using this knowledge base might not be -indicated. This is because it is not possible (or at least not trivial) -to guess that the call to ``kb(e.upper()+"journal"+"s")`` in a format -element is equal to a call to ``kb("Ejournals")``. You should then not -completely rely on this indication. - -Why are some format elements defined in field table omitted in the format element documentation? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Some format elements defined in the "Logical Fields" management -interface of the BibIndex module (the basic format elements) are not -shown in the format elements documentation pages. We do not show such an -element if its name starts with a number. This is to reduce the number -of elements shown in the documentation as the logical fields table -contains a lot of not so useful fields to be used in templates. - -How can I get access to repeatable subfields from inside a format element? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Given that repeatable subfields are not frequent, the ``bfo.fields(..)`` -function has been implemented to return the most convenient structure -for most cases, that is a '*list of strings*\ ' (**Case 1** below) or -'*list of dict of strings*\ ' (**Case 2** below). For eg. with the -following metadata: - -:: - - 999C5 $a value_1a $b value_1b - 999C5 $b value_2b - 999C5 $b value_3b $b value_3b_bis - - >> bfo.fields('999C5b') (1) - >> ['value_1b', 'value_2b', 'value_3b', 'value_3b_bis'] - >> bfo.fields('999C5') (2) - >> [{'a':'value_1a', 'b':'value_1b'}, - {'b':'value_2b'}, - {'b':'value_3b'}] - -In this example ``value3b_bis`` is not shown for -``bfo.fields('999C5')`` (**Case 2**). If it were to be taken into -account, the returned structure would have to be a '*list of dict of -list of strings*\ ', thus making for most cases the access to the data a -bit more complex. - -In order to consider the repeatable subfields, use the additional -``repeatable_subfields_p`` parameter: - -:: - - >> bfo.fields('999C5b', repeatable_subfields_p=True) (1 bis) - >> ['value_1b', 'value_2b', 'value_3b'] - >> bfo.fields('999C5', repeatable_subfields_p=True) (2 bis) - >> [{'a':['value_1a'], 'b':['value_1b']}, - {'b':['value_2b']}, - {'b':['value_3b', 'value3b_bis']}] - -Another solution would be to access the BibRecord structure with -``bfo.getRecord()`` and use the lower-level BibRecord module with this -structure. - -.. |Use output format HD| image:: /_static/admin/bibformat-guide-url_bar.png -.. |BibFormat Guide BFO HD rules| image:: /_static/admin/bibformat-guide-bfo_hd_rules.png -.. |Output formats management page| image:: /_static/admin/bibformat-guide-bfo_manage.png -.. |Screenshot of the Update Output Format Attributes page| image:: /_static/admin/bibformat-guide-bfo_attributes.png -.. |Output format menu| image:: /_static/admin/bibformat-guide-bfo_rules.png -.. |Format template management page| image:: /_static/admin/bibformat-guide-bft_manage.png -.. |update format template attributes| image:: /_static/admin/bibformat-guide-bft_attributes.png -.. |Format template editor| image:: /_static/admin/bibformat-guide-bft_editor2.png -.. |Rule: Use template Picture HTML Detailed if field 980__a is equal to PICTURE| image:: /_static/admin/bibformat-guide-bfo_edit_rule.png -.. |Reorder rules using arrows| image:: /_static/admin/bibformat-guide-bfo_edit_rule2.png -.. |Check Dependencies menu| image:: /_static/admin/bibformat-guide-bfo_check_deps.png -.. |Template Editor Page| image:: /_static/admin/bibformat-guide-bft_editor.png -.. |Preview Panel| image:: /_static/admin/bibformat-guide-bft_preview.png -.. |Image representation of HB output format| image:: /_static/admin/bibformat-guide-bfo_hb_migrate.png -.. |bibformat.gif| image:: /_static/admin/bibformat-guide-bfe.gif diff --git a/docs/admin/index.rst b/docs/admin/index.rst index ad85ca35d..8031f5284 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -1,55 +1,54 @@ .. This file is part of Invenio Copyright (C) 2014 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. .. _admins-guide: ============= Admin's Guide ============= .. toctree:: :maxdepth: 1 bibauthority-admin-guide bibcheck-admin-guide bibcirculation-admin-guide bibclassify-admin-guide bibconvert-admin-guide bibedit-admin-guide - bibformat-admin-guide bibindex-admin-guide bibknowledge-admin-guide bibmatch-admin-guide bibrank-admin-guide bibsched-admin-guide bibsort-admin-guide bibsword-client-admin-guide bibupload-admin-guide docextract-admin-guide elmsubmit-admin-guide oaiharvest-admin-guide oairepository-admin-guide publiline-admin-guide webcomment-admin-guide webjournal-admin-guide webjournal-editor-guide webmessage-admin-guide websearch-admin-guide websession-admin-guide webstat-admin-guide webstyle-admin-guide websubmit-admin-guide diff --git a/docs/admin/webstyle-admin-guide.rst b/docs/admin/webstyle-admin-guide.rst index c242f918a..ac58ee1ea 100644 --- a/docs/admin/webstyle-admin-guide.rst +++ b/docs/admin/webstyle-admin-guide.rst @@ -1,517 +1,515 @@ .. This file is part of Invenio Copyright (C) 2014 CERN. Invenio is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Invenio is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with Invenio; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. .. _webstyle-admin-guide: WebStyle Admin Guide ==================== Contents -------- - **1. `Overview <#overview_page_layout>`__** - 1.1  \ `CSS Style Sheet and Images <#overview_css>`__ - 1.2  \ `HTML Page Layout <#overview_page_layout>`__ - 1.2.1    \ `Layout of HTML Static Pages <#overview_page_layout_stat>`__ - 1.2.2    \ `Layout of Python Dynamic Pages <#overview_page_layout_dyn>`__ - 1.3  \ `Look of Bibliographic References <#overview_bib>`__ - 1.4  \ `Specific Configurations <#overview_spec_conf>`__ - **2. `Detailed Record Pages <#det_page>`__** - 2.1  \ `Available tabs <#det_page>`__ - 2.2  \ `Showing/Hiding tabs <#det_show_hide_tabs>`__ - 2.3  \ `Customizing content of tabs <#det_page_cust_cont_tabs>`__ - 2.4  \ `Customizing look of tabs <#det_page_cust_look_tabs>`__ - **3. `Custom Redirections <#red>`__** - 3.1  \ `Command Line Interface <#red_cli>`__ 1. Overview ----------- This document describes how to change the look and feel of your CDS Invenio installation. 1.1 CSS Style Sheet and Images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most obvious modification you may want to do is the modification of `CSS style sheet `__. You may also customize default `images `__. 1.2 HTML Page Layout ~~~~~~~~~~~~~~~~~~~~ The customization of the general page look and feel is currently different depending on whether you customize HTML-like static pages or dynamic Python pages. Dynamic HTML pages are used to build the 'interactive' parts of the website, such as the search and browse pages, as well as the admin pages. The content of these pages is defined at run time, depending on the users parameters. You can modify them to provide a totally different experience to your users when browsing Atlantis Institute of Fictive Science. Most probably, you will only want to customize 'webstyle\_templates.py', which define the headers and footer of a page. Static HTML pages are used for basic pages that do not embed dynamic content, such as this guide. They help reducing the load of the server and speed up pages serving. As you will see, even HTML static pages can still contain some values defined at run time, but these are reduced to the minimum compared to dynamic pages. Most probably you will not want to modify these pages, since the small amount of dynamic content they have allow these pages to inherit from the customizations you have made elsewhere, such as in the CSS, and the page header and footer. 1.2.1 Layout of HTML Static Pages ````````````````````````````````` Static HTML pages are all located in the /opt/invenio/lib/webdoc/ installation directory. These files are organized in 3 directories: - **help:** Help pages available to users of your website - **admin:** Mostly guides for admin users - **hacking:** Mostly guides for administrators and developers These directories do not contain the ``.html`` files, but ``.webdoc`` files. These '*WebDoc*\ ' files are basically HTML with the following main differences: - They only contain the body of your page (only content of the ```` tag) - You can make use of special tags such as ```` and ````, that will be replaced by http://localhost:4000 and info@invenio-software.org, the values that you should have configured in your ``config.py`` file. - You can internationalize the content. For example: :: Book Livre Buch will be replaced by "Book" if the user has chosen to display the English version of your site, "Livre" in French or "Buch" in German. Read the `WebDoc syntax guide `__ to learn more about details of the WebDoc syntax. The advantage of not using raw HTML for these static pages is that they can for example reuse the header and footer that you have defined for dynamic pages, and make use of the variables you have defined in your invenio.conf file. In that way you should not need to adapt them to your needs: **they** will adapt themselves to your needs. Any modification should be immediatly visible when looking at the pages from the web: the pages are built "dynamically" and then cached. If you want to force the cache of the pages, use the WebDoc CLI (type ``/opt/invenio/bin/webdoc --help`` to learn more about it). 1.2.2 Layout of Python Dynamic Pages ```````````````````````````````````` The dynamic Python-powered pages can be customized by making use of Invenio templating system that uses a notion of a template skin. How this works? When you edit ``invenio-local.conf`` during installation or later during runtime (when during runtime, you have to run ``inveniocfg --update-config-py`` after editing the conf file), you may choose to use your own templates instead of the provided default ones by editing ``CFG_WEBSTYLE_TEMPLATE_SKIN`` variable. Let us say you put ``ithaca`` there in order to use your own ``ithaca`` style. Now, when you start Apache, then instead of Invenio's usual template files such as ``webbasket_templates.py`` the system will look for file named ``webbasket_templates_ithaca.py`` and will load the template functions from there instead, provided that they exist. (Otherwise it would fall back to the default ones.) How do you create such an ``ithaca`` style templates file? We do not use one of many existing templating frameworks in Python but a very simple programmer-friendly templating system that enables you to use the full power of Python to inherit from the default templates the output generating functions you want to reuse and to write anew only the functions you would like to modify. Let's show an example of how to modify the page footer. Create a file named ``webstyle_templates_ithaca.py`` with the following content: :: from invenio.config import CFG_SITE_LANG from invenio.legacy.webstyle.templates import Template as DefaultTemplate class Template(DefaultTemplate): """Ithaca style templates.""" def tmpl_pagefooter(self, req=None, ln=CFG_SITE_LANG, lastupdated=None, pagefooteradd=""): """ Ithaca style page footer. See the default function for the meaning of parameters. """ out = "" out += """
This site has no footer. """ return out After the file was created, restart Apache and lo, your new ithaca style footer will be seen in action. (A side comment: note that ``tmpl_page_footer()`` is an ideal place to put any local code you may want to execute at the end of web request processing when the main page content was just served to the user. As an example, if you are using Google Analytics, you may want to put just after the above ``out = ""`` statement your GA script code: :: [...] out += """ """ End of the side comment.) Some further remarks on this templating system: - We have observed that in practice the HTML page designers were ofter Python programmers, therefore we have adopted a programmer-friendly templating system. - You have to know a bit of Python in order to use it. If you don't know Python, do not worry, because you can basically copy and paste the original ``tmpl_foo()`` function definition "as is" into the above-cited example and then you would only modify its HTML snippets. The important thing is to preserve the imports (``from invenio.config import CFG_SITE_LANG``) as in the original ``webstyle_templates.py`` file and to preserve the leading whitespace Pythonic indentation. - You do not have to learn "yet another templating language", you can use the full power of Python. The ``tmpl_foo()`` functions do not contain any business logic in them, their only purpose is to make the HTML presentation of data supplied to them. But, should you need to carry out a little data transformation, you can do it within the ``tmpl_foo()`` function itself, thanks to the full Python power. - If you feel like doing so, you can modify all the ``tmpl_foo()`` functions across all Invenio modules in a way that will completely change the presentation of elements including their content, position and order on the screen. - In practice, it is sufficient to modify the CSS and the webstyle\_templates\_ithaca.py (and possibly websearch\_templates\_ithaca.py) files to achieve most important customizations. - If you would like to discover which method of which template generate which region on the web page, you can switch on the ``CFG_WEBSTYLE_INSPECT_TEMPLATES`` configuration variable in your ``invenio-local.conf`` file and rerun ``sudo -u apache /opt/invenio/bin/inveniocfg --update-config-py``. Then, after optionally running ``bibreformat -a`` and ``webcoll -f`` (if you want to debug search pages) and after having restarted your Apache server (in every case), you will find in your browser that a place-mark has been put next to every region of every page, and that you can hover your mouse pointer over any region of the page in order to discover which module/method/parameters have been used to generate it. This is useful for debugging Python templates and/or for understanding which part of code generates which HTML snippet in the output. - We expect to provide possibly more than one skin with the default distribution, so if you have modified Invenio look and feel in an interesting way, please consider donating us your templates. - When upgrading from one Invenio release to another, you may find out that the default templates have changed in a way that requires changes to your templates (such as an addition of parameters to cover the new functionality). This is inevitable in any templating system; unless you introduce new parameters, you would not see them being printed. Therefore, if you have modified ``tmpl_foo()`` and ``tmpl_bar()``, and you are ugrading to a new release, you may at least briefly check whether the function arguments are the same. A quick check of the body would be helpful too, in case the new release fixed some display-related problems in these functions. In order to help you in this task, we provide a tool to check incompatibilities between your customized templates and the default templates. This tool can be run before doing a ``'make install'``, therefore giving you a chance to fix your templates before upgrading. Just run ``'make check-custom-templates'`` to get the list of problems found with your templates. You can also run this tool any time after the new default templates have been installed, in order to ensure that modifications you have done to your templates are valid. To do so move to your Invenio installation directory, and run::: $ python /opt/invenio/lib/python/invenio/template.py --check-custom-templates 1.3 Look of Bibliographic References ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Bibliographic metadata is formatted using -`BibFormat `__. Read the `BibFormat -documentation `__ for more information. +Bibliographic metadata is formatted using ``formatter`` module. 1.4 Specific Configurations ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that the search interface pages may be modified to a large extent in the `WebSearch Admin Interface `__ by adding HTML portalboxes on various places on the page (right top, before/after page title, before/after narrow by collection boxes, etc). 2. Detailed Record Pages ------------------------ The web pages displaying the details of a record (such as /record/1) do not only show metadata, but also users' comments and reviews, statistics, etc. This information is organized into tabs. The content of these tabs can be customized on a collection basis. It is also possible to show/hide tabs depending on the displayed collection. The detailed record pages also feature a mini panel at the bottom of the page that links to popular functions (The mini panel is only displayed when *Information* tab is selected). :: +--------------Detailed record page-------------+ | header | |nav. breadcrumb | | | | .--------------------------------------. | | .-|Info.|Ref.|Comm.|Review.|Stats.|Files |-. | | | '--------------------------------------' | | | | | | | | content | | | | | | | '------------------------------------------' | | | | .---------------(Mini Panel)---------------. | | | Mini | Mini | Mini | | | | File | Review | Actions | | | '------------------------------------------' | +-----------------------------------------------+ 2.1 Available tabs ~~~~~~~~~~~~~~~~~~ The following tabs are available: +-------------------------+--------------------------+-----------------------+ | Name | | | | Description | | | | URL (eg. for record | | | | '10') | | | +=========================+==========================+=======================+ | Information | References | Comments | | Show the formatted | Displays the references | Displays the users' | | metadata of the record | (bibliography) of the | comments | | /re | record | http://localhost:4000/re | | | cord/10 | /re \| cord/10/comments | | | | cord/10/references | | +-------------------------+--------------------------+-----------------------+ The mini panel is only displayed when the *Information* tab is selected. It is divided into the following sections: - Files: quick access to full-text file(s) - Review: quick access to reviewing feature - Actions: quick access to several other features 2.2 Showing/Hiding tabs ~~~~~~~~~~~~~~~~~~~~~~~ The `WebSearch admin web interface `__ lets you decide for each collection which tabs are to be displayed. Choose a collection to edit in the collection tree and go to its *detailed record page options*. From there you can select which tabs to show for that collection. If you want to apply these settings to the subcollections, select *Also apply to subcollections* before you click on the button. Note that these settings only affect the tabs, not the content of the tabs: even if a tab is not displayed, it is still possible to access its content using its usual url. This is useful if you decide to completely change the detailed record pages, dropping the tab-metaphor (eg. for a side bar) but still want to access the comments, reviews, etc pages. Here are some behaviours you should expect when changing the tabs configuration: - Given that search results pages always link to /record/10, and given the above comment about accessibility of tabs when they are not displayed, the content of the *Information* will always be show when clicking on `detailed record <#>`__ link in search results, even if the *Information* tab is set not to be displayed. - If you select only 1 tab, none of the tabs will be displayed at the top of the page. This also means that whatever tabs you have selected, you users will always see the content of the 'Information' tabs (see above behaviour). - If you select 0 tab, only the content of *Information* tab is shown. None of the tabs, nor the border that usually surrounds the content of the tabs, nor the minipanel are shown. You should choose this option if you decide to drop the tabs metaphor for the detailed record pages. You can then build your own user interface on this almost blank page (See `Customizing content of tabs <#det_page_cust_cont_tabs>`__). - Note that *Comments* and *Reviews* tabs will not be shown if you have disabled commenting and reviewing features in your installation, respectively. (``CFG_WEBCOMMENT_ALLOW_COMMENTS`` and ``CFG_WEBCOMMENT_ALLOW_REVIEWS`` variable in your config file) 2.3 Customizing content of tabs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The contents of tabs are defined in the following ways: ***Information* tab** The content of this tab is defined by function ``tmpl_detailed_record_metadata(..)`` in ``websearch_templates.py``. By default ``tmpl_detailed_record_metadata`` simply returns the result of the formatting of the metadata by BibFormat using the "HD" output format. It can therefore be collection-specific. ***References* tab** The content of this tab is defined by function ``tmpl_detailed_record_references(..)`` in ``websearch_templates.py``. By default ``tmpl_detailed_record_metadata`` simply returns the result of the formatting of the metadata by BibFormat using the "HDREF" output format. If the result returned by BibFormat is empty, the tab is disabled (visible, but not clickable). It can therefore be collection-specific. ***Comments* and *Reviews* tabs** The content of these tabs is mainly defined by function ``tmpl_get_comments(..)`` in ``webcomment_templates.py``. Other functions in this file are also involved in the display. ***Usage Statistics* tab** The content of this tab is defined by function ``tmpl_detailed_record_statistics(..)`` in ``websearch_templates.py``. If the returned content is empty, then the tabs will be disabled (visible, but cannot be clicked). ***Files* tab** The content of this tab is defined by function ``tmpl_filelist(..)`` in ``websubmit_templates.py``. The content of the mini panel is defined in the following ways: ***Files*** The content of this section is defined by the output format 'HDFILE'. It can therefore be collection-specific. ***Review*** The content of this section is defined by function ``tmpl_mini_review(..)`` inside ``webcomment_templates.py`` ***Actions*** The content of this section is defined by the output format ``HDACT``. It can therefore be collection-specific. 2.4 Customizing look of tabs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can customize how tabs look like, as well change the look of the border that surrounds the content of tabs. The mini panel can similarly be customized. Have a look at the following classes in the CDS css stylesheet: - ``detailedrecordtabs`` - ``detailedrecordbox`` - ``detailedrecordminipanel`` - ``top-left, top-right, bottom-left, bottom-right`` - ``detailedrecordminipanel{actions,review,file}, detailedrecordshortreminder`` Note that a tab might be greyed out (disabled) when its content is empty. This is the case for the *References* tab (see `Customizing content of tabs <#det_page_cust_cont_tabs>`__ -> 'References tab') and the *Files* tab (if no file could be found for the record). For more advanced modifications (like changing the HTML code of the tabs), you can modify the ``detailed_record_container(..)`` and ``detailed_record_mini_panel(..)`` functions inside your ``webstyle_templates.py`` file. Custom Redirections ~~~~~~~~~~~~~~~~~~~ It is possible to create custom redirections to URLs within Invenio, by registering a given *unique label* to be used after path **/goto/**. FIXME Command Line Interface ~~~~~~~~~~~~~~~~~~~~~~ :: Usage: gotoadmin [options] Options: -h, --help show this help message and exit Plugin Administration Options: --list-plugins List available GOTO plugins and their documentation --list-broken-plugins List broken GOTO plugins Redirection Manipultation Options: -r LABEL, --register-redirection=LABEL Register a redirection with the provided LABEL -u LABEL, --update-redirection=LABEL Update the redirection specified by the provided LABEL -g LABEL, --get-redirection=LABEL Get all information about a redirection specified by LABEL -d LABEL, --drop-redirection=LABEL Drop an existing redirection specified by LABEL Specific Options: -P PLUGIN, --plugin=PLUGIN Specify the plugin to use when registering or updating a redirection -j PARAMETERS, --json-parameters=PARAMETERS Specify the parameters to provide to the plugin (serialized in JSON) -p PARAM=VALUE, --parameter=PARAM=VALUE Specify a single PARAM=VALUE parameter to be provided to the plugin (alternative to the JSON serialization) diff --git a/invenio/base/config.py b/invenio/base/config.py index 6a427e339..2197fb6dd 100644 --- a/invenio/base/config.py +++ b/invenio/base/config.py @@ -1,755 +1,743 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2012, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Default configuration values.""" from __future__ import unicode_literals import distutils.sysconfig from os.path import join -from werkzeug import LocalProxy - -from invenio.utils.deprecation import deprecated, RemovedInInvenio22Warning from invenio.utils.shell import which from invenio.version import __version__ EXTENSIONS = [ 'invenio.ext.confighacks', - 'invenio.ext.jinja2hacks', 'invenio.ext.passlib:Passlib', 'invenio.ext.debug_toolbar', 'invenio.ext.babel', 'invenio.ext.sqlalchemy', 'invenio.ext.sslify', 'invenio.ext.cache', 'invenio.ext.session', 'invenio.ext.login', 'invenio.ext.principal', 'invenio.ext.email', 'invenio.ext.fixtures', # before legacy 'invenio.ext.legacy', 'invenio.ext.assets', 'invenio.ext.template', 'invenio.ext.admin', 'invenio.ext.logging', 'invenio.ext.logging.backends.fs', 'invenio.ext.logging.backends.legacy', 'invenio.ext.logging.backends.sentry', 'invenio.ext.gravatar', 'invenio.ext.collect', 'invenio.ext.restful', 'invenio.ext.menu', 'invenio.ext.jasmine', # after assets 'flask_breadcrumbs:Breadcrumbs', 'invenio.modules.deposit.url_converters', 'invenio.ext.iiif', ] PACKAGES = [ 'invenio.modules.*', 'invenio.base', ] PACKAGES_EXCLUDE = [ 'invenio.modules.annotations', 'invenio.modules.archiver', 'invenio.modules.communities', # remove with invenio/modules/communities 'invenio.modules.linkbacks', 'invenio.modules.multimedia', 'invenio.modules.pages', ] LEGACY_WEBINTERFACE_EXCLUDE = [ 'invenio.legacy.websubmit', ] _cfg_prefix = distutils.sysconfig.get_config_var("prefix") - -@deprecated( - 'Use a more specific variable from invenio/base/config.py instead', - RemovedInInvenio22Warning) -def _cfg_prefix_for_proxy(): - return _cfg_prefix - -CFG_PREFIX = LocalProxy(_cfg_prefix_for_proxy) CFG_DATADIR = join(_cfg_prefix, 'var', 'data') CFG_BATCHUPLOADER_DAEMON_DIR = join(_cfg_prefix, "var", "batchupload") CFG_BATCHUPLOADER_DAEMON_DIR = CFG_BATCHUPLOADER_DAEMON_DIR[0] == '/' and CFG_BATCHUPLOADER_DAEMON_DIR \ or _cfg_prefix + '/' + CFG_BATCHUPLOADER_DAEMON_DIR CFG_BIBDOCFILE_FILEDIR = join(CFG_DATADIR, "files") CFG_BINDIR = join(_cfg_prefix, "bin") CFG_ETCDIR = join(_cfg_prefix, "etc") CFG_CACHEDIR = join(_cfg_prefix, "var", "cache") CFG_LOGDIR = join(_cfg_prefix, "var", "log") CFG_RUNDIR = join(_cfg_prefix, "var", "run") CFG_TMPDIR = join(_cfg_prefix, "var", "tmp") CFG_WEBDIR = join(_cfg_prefix, "var", "www") CFG_PYLIBDIR = join(_cfg_prefix, "lib", "python") CFG_LOCALEDIR = join(_cfg_prefix, "share", "locale") CFG_TMPSHAREDDIR = join(_cfg_prefix, "var", "tmp-shared") CFG_WEBSUBMIT_BIBCONVERTCONFIGDIR = join(_cfg_prefix, "etc", "bibconvert", "config") CFG_WEBSUBMIT_COUNTERSDIR = join(CFG_DATADIR, "submit", "counters") CFG_WEBSUBMIT_STORAGEDIR = join(CFG_DATADIR, "submit", "storage") CFG_BIBEDIT_CACHEDIR = join(_cfg_prefix, "var", "tmp-shared", "bibedit-cache") CFG_COMMENTSDIR = join(CFG_DATADIR, "comments") CFG_BASKETSDIR = join(CFG_DATADIR, "baskets") # FIXME check the usage and replace by SQLALCHEMY_URL CFG_DATABASE_HOST = "localhost" CFG_DATABASE_NAME = "invenio" CFG_DATABASE_PASS = "my123p$ss" CFG_DATABASE_PORT = 3306 CFG_DATABASE_SLAVE = None CFG_DATABASE_TYPE = "mysql" CFG_DATABASE_USER = "invenio" # CFG_FLASK_CACHE_TYPE has been deprecated. CACHE_TYPE = "redis" REQUIREJS_CONFIG = "js/build.js" # DO NOT EDIT THIS FILE! IT WAS AUTOMATICALLY GENERATED # FROM INVENIO.CONF BY EXECUTING: # inveniocfg --update-all CFG_SITE_NAME_INTL = {} CFG_SITE_NAME_INTL['af'] = "Atlantis Instituut van Fiktiewe Wetenskap" CFG_SITE_NAME_INTL['ar'] = "معهد أطلنطيس للعلوم الافتراضية" CFG_SITE_NAME_INTL['bg'] = "Институт за фиктивни науки Атлантис" CFG_SITE_NAME_INTL['ca'] = "Institut Atlantis de Ciència Fictícia" CFG_SITE_NAME_INTL['cs'] = "Atlantis Institut Fiktivních Věd" CFG_SITE_NAME_INTL['de'] = "Atlantis Institut der fiktiven Wissenschaft" CFG_SITE_NAME_INTL['el'] = "Ινστιτούτο Φανταστικών Επιστημών Ατλαντίδος" CFG_SITE_NAME_INTL['en'] = "Atlantis Institute of Fictive Science" CFG_SITE_NAME_INTL['es'] = "Atlantis Instituto de la Ciencia Fictive" CFG_SITE_NAME_INTL['fr'] = "Atlantis Institut des Sciences Fictives" CFG_SITE_NAME_INTL['hr'] = "Institut Fiktivnih Znanosti Atlantis" CFG_SITE_NAME_INTL['gl'] = "Instituto Atlantis de Ciencia Fictive" CFG_SITE_NAME_INTL['ka'] = "ატლანტიდის ფიქტიური მეცნიერების ინსტიტუტი" CFG_SITE_NAME_INTL['it'] = "Atlantis Istituto di Scienza Fittizia" CFG_SITE_NAME_INTL['rw'] = "Atlantis Ishuri Rikuru Ry'ubuhanga" CFG_SITE_NAME_INTL['lt'] = "Fiktyvių Mokslų Institutas Atlantis" CFG_SITE_NAME_INTL['hu'] = "Kitalált Tudományok Atlantiszi Intézete" CFG_SITE_NAME_INTL['ja'] = "Fictive 科学のAtlantis の協会" CFG_SITE_NAME_INTL['no'] = "Atlantis Institutt for Fiktiv Vitenskap" CFG_SITE_NAME_INTL['pl'] = "Instytut Fikcyjnej Nauki Atlantis" CFG_SITE_NAME_INTL['pt'] = "Instituto Atlantis de Ciência Fictícia" CFG_SITE_NAME_INTL['ro'] = "Institutul Atlantis al Ştiinţelor Fictive" CFG_SITE_NAME_INTL['ru'] = "Институт Фиктивных Наук Атлантиды" CFG_SITE_NAME_INTL['sk'] = "Atlantis Inštitút Fiktívnych Vied" CFG_SITE_NAME_INTL['sv'] = "Atlantis Institut för Fiktiv Vetenskap" CFG_SITE_NAME_INTL['uk'] = "Інститут вигаданих наук в Атлантісі" CFG_SITE_NAME_INTL['zh_CN'] = "阿特兰提斯虚拟科学学院" CFG_SITE_NAME_INTL['zh_TW'] = "阿特蘭提斯虛擬科學學院" CFG_ACCESS_CONTROL_LEVEL_ACCOUNTS = 0 CFG_ACCESS_CONTROL_LEVEL_GUESTS = 0 CFG_ACCESS_CONTROL_LEVEL_SITE = 0 CFG_ACCESS_CONTROL_LIMIT_REGISTRATION_TO_DOMAIN = "" CFG_ACCESS_CONTROL_NOTIFY_ADMIN_ABOUT_NEW_ACCOUNTS = 0 CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_ACTIVATION = 0 CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_DELETION = 0 CFG_ACCESS_CONTROL_NOTIFY_USER_ABOUT_NEW_ACCOUNT = 1 CFG_ADS_SITE = 0 CFG_APACHE_GROUP_FILE = "demo-site-apache-user-groups" CFG_APACHE_PASSWORD_FILE = "demo-site-apache-user-passwords" CFG_ARXIV_URL_PATTERN = "http://export.arxiv.org/pdf/%sv%s.pdf" CFG_BATCHUPLOADER_FILENAME_MATCHING_POLICY = ['reportnumber', 'recid', ] CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS = r"invenio_webupload|Invenio-.*" CFG_BATCHUPLOADER_WEB_ROBOT_RIGHTS = { '127.0.0.1': ['*'], # useful for testing '127.0.1.1': ['*'], # useful for testing '10.0.0.1': ['BOOK', 'REPORT'], # Example 1 '10.0.0.2': ['POETRY', 'PREPRINT'], # Example 2 } CFG_BIBAUTHORID_AUTHOR_TICKET_ADMIN_EMAIL = "info@invenio-software.org" CFG_BIBAUTHORID_ENABLED = True CFG_BIBAUTHORID_EXTERNAL_CLAIMED_RECORDS_KEY = [] CFG_BIBAUTHORID_MAX_PROCESSES = 12 CFG_BIBAUTHORID_ON_AUTHORPAGES = True CFG_BIBAUTHORID_PERSONID_SQL_MAX_THREADS = 12 CFG_BIBAUTHORID_UI_SKIP_ARXIV_STUB_PAGE = False CFG_BIBAUTHORID_SEARCH_ENGINE_MAX_DATACHUNK_PER_INSERT_DB_QUERY = 10000000 CFG_BIBCATALOG_SYSTEM = "EMAIL" CFG_BIBCATALOG_SYSTEM_EMAIL_ADDRESS = "info@invenio-software.org" CFG_BIBCATALOG_SYSTEM_RT_CLI = "/usr/bin/rt" CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_PWD = "" CFG_BIBCATALOG_SYSTEM_RT_DEFAULT_USER = "" CFG_BIBCATALOG_SYSTEM_RT_URL = "http://localhost/rt3" CFG_BIBCIRCULATION_ACQ_STATUS_CANCELLED = "cancelled" CFG_BIBCIRCULATION_ACQ_STATUS_NEW = "new" CFG_BIBCIRCULATION_ACQ_STATUS_ON_ORDER = "on order" CFG_BIBCIRCULATION_ACQ_STATUS_PARTIAL_RECEIPT = "partial receipt" CFG_BIBCIRCULATION_ACQ_STATUS_RECEIVED = "received" CFG_BIBCIRCULATION_AMAZON_ACCESS_KEY = "" CFG_BIBCIRCULATION_ILL_STATUS_CANCELLED = "cancelled" CFG_BIBCIRCULATION_ILL_STATUS_NEW = "new" CFG_BIBCIRCULATION_ILL_STATUS_ON_LOAN = "on loan" CFG_BIBCIRCULATION_ILL_STATUS_RECEIVED = "received" CFG_BIBCIRCULATION_ILL_STATUS_REQUESTED = "requested" CFG_BIBCIRCULATION_ILL_STATUS_RETURNED = "returned" CFG_BIBCIRCULATION_ITEM_STATUS_CANCELLED = "cancelled" CFG_BIBCIRCULATION_ITEM_STATUS_CLAIMED = "claimed" CFG_BIBCIRCULATION_ITEM_STATUS_IN_PROCESS = "in process" CFG_BIBCIRCULATION_ITEM_STATUS_NOT_ARRIVED = "not arrived" CFG_BIBCIRCULATION_ITEM_STATUS_ON_LOAN = "on loan" CFG_BIBCIRCULATION_ITEM_STATUS_ON_ORDER = "on order" CFG_BIBCIRCULATION_ITEM_STATUS_ON_SHELF = "on shelf" CFG_BIBCIRCULATION_ITEM_STATUS_OPTIONAL = [] CFG_BIBCIRCULATION_ITEM_STATUS_UNDER_REVIEW = "under review" CFG_BIBCIRCULATION_LIBRARY_TYPE_EXTERNAL = "external" CFG_BIBCIRCULATION_LIBRARY_TYPE_HIDDEN = "hidden" CFG_BIBCIRCULATION_LIBRARY_TYPE_INTERNAL = "internal" CFG_BIBCIRCULATION_LIBRARY_TYPE_MAIN = "main" CFG_BIBCIRCULATION_LOAN_STATUS_EXPIRED = "expired" CFG_BIBCIRCULATION_LOAN_STATUS_ON_LOAN = "on loan" CFG_BIBCIRCULATION_LOAN_STATUS_RETURNED = "returned" CFG_BIBCIRCULATION_PROPOSAL_STATUS_NEW = "proposal-new" CFG_BIBCIRCULATION_PROPOSAL_STATUS_ON_ORDER = "proposal-on order" CFG_BIBCIRCULATION_PROPOSAL_STATUS_PUT_ASIDE = "proposal-put aside" CFG_BIBCIRCULATION_PROPOSAL_STATUS_RECEIVED = "proposal-received" CFG_BIBCIRCULATION_REQUEST_STATUS_CANCELLED = "cancelled" CFG_BIBCIRCULATION_REQUEST_STATUS_DONE = "done" CFG_BIBCIRCULATION_REQUEST_STATUS_PENDING = "pending" CFG_BIBCIRCULATION_REQUEST_STATUS_PROPOSED = "proposed" CFG_BIBCIRCULATION_REQUEST_STATUS_WAITING = "waiting" CFG_BIBCLASSIFY_WEB_MAXKW = 100 CFG_BIBDOCFILE_ADDITIONAL_KNOWN_FILE_EXTENSIONS = [ 'hpg', 'link', 'lis', 'llb', 'mat', 'mpp', 'msg', 'docx', 'docm', 'xlsx', 'xlsm', 'xlsb', 'pptx', 'pptm', 'ppsx', 'ppsm', ] CFG_BIBDOCFILE_ADDITIONAL_KNOWN_MIMETYPES = { "application/xml-dtd": ".dtd", } CFG_BIBDOCFILE_BEST_FORMATS_TO_EXTRACT_TEXT_FROM = ( 'txt', 'html', 'xml', 'odt', 'doc', 'docx', 'djvu', 'pdf', 'ps', 'ps.gz') CFG_BIBDOCFILE_DESIRED_CONVERSIONS = { 'pdf': ('pdf;pdfa', ), 'ps.gz': ('pdf;pdfa', ), 'djvu': ('pdf', ), 'sxw': ('doc', 'odt', 'pdf;pdfa', ), 'docx': ('doc', 'odt', 'pdf;pdfa', ), 'doc': ('odt', 'pdf;pdfa', 'docx'), 'rtf': ('pdf;pdfa', 'odt', ), 'odt': ('pdf;pdfa', 'doc', ), 'pptx': ('ppt', 'odp', 'pdf;pdfa', ), 'ppt': ('odp', 'pdf;pdfa', 'pptx'), 'sxi': ('odp', 'pdf;pdfa', ), 'odp': ('pdf;pdfa', 'ppt', ), 'xlsx': ('xls', 'ods', 'csv'), 'xls': ('ods', 'csv'), 'ods': ('xls', 'xlsx', 'csv'), 'sxc': ('xls', 'xlsx', 'csv'), 'tiff': ('pdf;pdfa', ), 'tif': ('pdf;pdfa', ), } CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_DOCTYPES = [ ('Main', 'Main document'), ('LaTeX', 'LaTeX'), ('Source', 'Source'), ('Additional', 'Additional File'), ('Audio', 'Audio file'), ('Video', 'Video file'), ('Script', 'Script'), ('Data', 'Data'), ('Figure', 'Figure'), ('Schema', 'Schema'), ('Graph', 'Graph'), ('Image', 'Image'), ('Drawing', 'Drawing'), ('Slides', 'Slides')] CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_MISC = { 'can_revise_doctypes': ['*'], 'can_comment_doctypes': ['*'], 'can_describe_doctypes': ['*'], 'can_delete_doctypes': ['*'], 'can_keep_doctypes': ['*'], 'can_rename_doctypes': ['*'], 'can_add_format_to_doctypes': ['*'], 'can_restrict_doctypes': ['*'], } CFG_BIBDOCFILE_DOCUMENT_FILE_MANAGER_RESTRICTIONS = [ ('', 'Public'), ('restricted', 'Restricted')] CFG_BIBDOCFILE_ENABLE_BIBDOCFSINFO_CACHE = 0 CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT = 5000 CFG_BIBDOCFILE_MD5_CHECK_PROBABILITY = 0.1 CFG_BIBDOCFILE_USE_XSENDFILE = 0 CFG_BIBDOCFILE_AFS_VOLUME_PATTERN = "p.invenio.%s" CFG_BIBDOCFILE_AFS_VOLUME_QUOTA = 10000000 CFG_BIBDOCFILE_PREFERRED_MIMETYPES_MAPPING = { 'application/msword': '.doc', 'application/octet-stream': '.bin', 'application/postscript': '.ps', 'application/vnd.ms-excel': '.xls', 'application/vnd.ms-powerpoint': '.ppt', 'application/x-gtar-compressed': '.tgz', 'application/xhtml+xml': '.xhtml', 'application/xml': '.xml', 'audio/mpeg': '.mp3', 'audio/ogg': '.ogg', 'image/jpeg': '.jpeg', 'image/svg+xml': '.svg', 'image/tiff': '.tiff', 'message/rfc822': '.eml', 'text/calendar': '.ics', 'text/plain': '.txt', 'video/mpeg': '.mpeg', } CFG_BIBEDIT_EXTEND_RECORD_WITH_COLLECTION_TEMPLATE = {'POETRY': 'record_poem'} CFG_BIBEDIT_KB_SUBJECTS = "Subjects" CFG_BIBEDIT_LOCKLEVEL = 3 CFG_BIBEDIT_PROTECTED_FIELDS = "" CFG_BIBEDIT_QUEUE_CHECK_METHOD = "bibrecord" CFG_BIBEDIT_TIMEOUT = 3600 CFG_BIBEDIT_ADD_TICKET_RT_QUEUES = [] CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING = 20000 CFG_BIBEDITMULTI_LIMIT_DELAYED_PROCESSING_TIME = "22:00-05:00" CFG_BIBEDITMULTI_LIMIT_INSTANT_PROCESSING = 2000 CFG_BIBFIELD_MASTER_FORMATS = ['marc', ] CFG_BIBFORMAT_ADDTHIS_ID = "" CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS = [] CFG_BIBFORMAT_HIDDEN_FILE_FORMATS = [] CFG_BIBFORMAT_HIDDEN_TAGS = ['595', ] CFG_BIBINDEX_AUTHOR_WORD_INDEX_EXCLUDE_FIRST_NAMES = False CFG_BIBINDEX_CHARS_ALPHANUMERIC_SEPARATORS = \ r"[\!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~]" CFG_BIBINDEX_CHARS_PUNCTUATION = r"[\.\,\:\;\?\!\"]" CFG_BIBINDEX_FULLTEXT_INDEX_LOCAL_FILES_ONLY = 1 CFG_BIBINDEX_MIN_WORD_LENGTH = 0 CFG_BIBINDEX_PATH_TO_STOPWORDS_FILE = "etc/bibrank/stopwords.kb" CFG_BIBINDEX_PERFORM_OCR_ON_DOCNAMES = r"scan-.*" CFG_BIBINDEX_REMOVE_HTML_MARKUP = 0 CFG_BIBINDEX_REMOVE_LATEX_MARKUP = 0 CFG_BIBINDEX_REMOVE_STOPWORDS = 0 CFG_BIBINDEX_SPLASH_PAGES = { "http://documents\.cern\.ch/setlink\?.*": ".*", "http://ilcagenda\.linearcollider\.org/subContributionDisplay\.py\?.*|" "http://ilcagenda\.linearcollider\.org/contributionDisplay\.py\?.*": "http://ilcagenda\.linearcollider\.org/getFile\.py/access\?.*|" "http://ilcagenda\.linearcollider\.org/materialDisplay\.py\?.*", } CFG_BIBINDEX_SYNONYM_KBRS = { 'global': ['INDEX-SYNONYM-TITLE', 'exact'], 'title': ['INDEX-SYNONYM-TITLE', 'exact'], } CFG_BIBINDEX_URLOPENER_PASSWORD = "mysuperpass" CFG_BIBINDEX_URLOPENER_USERNAME = "mysuperuser" CFG_BIBMATCH_FUZZY_EMPTY_RESULT_LIMIT = 1 CFG_BIBMATCH_FUZZY_MATCH_VALIDATION_LIMIT = 0.65 CFG_BIBMATCH_FUZZY_WORDLIMITS = { '100__a': 2, '245__a': 4 } CFG_BIBMATCH_LOCAL_SLEEPTIME = 0.0 CFG_BIBMATCH_MATCH_VALIDATION_RULESETS = [ ('default', [{'tags': '245__%,242__%', 'threshold': 0.8, 'compare_mode': 'lazy', 'match_mode': 'title', 'result_mode': 'normal'}, {'tags': '037__a,088__a', 'threshold': 1.0, 'compare_mode': 'lazy', 'match_mode': 'identifier', 'result_mode': 'final'}, {'tags': '100__a,700__a', 'threshold': 0.8, 'compare_mode': 'normal', 'match_mode': 'author', 'result_mode': 'normal'}, {'tags': '773__a', 'threshold': 1.0, 'compare_mode': 'lazy', 'match_mode': 'title', 'result_mode': 'normal'}]), ('980__ \$\$a(THESIS|Thesis)', [{'tags': '100__a', 'threshold': 0.8, 'compare_mode': 'strict', 'match_mode': 'author', 'result_mode': 'normal'}, {'tags': '700__a,701__a', 'threshold': 1.0, 'compare_mode': 'lazy', 'match_mode': 'author', 'result_mode': 'normal'}, {'tags': '100__a,700__a', 'threshold': 0.8, 'compare_mode': 'ignored', 'match_mode': 'author', 'result_mode': 'normal'}]), ('260__', [{'tags': '260__c', 'threshold': 0.8, 'compare_mode': 'lazy', 'match_mode': 'date', 'result_mode': 'normal'}]), ('0247_', [{'tags': '0247_a', 'threshold': 1.0, 'compare_mode': 'lazy', 'match_mode': 'identifier', 'result_mode': 'final'}]), ('020__', [{'tags': '020__a', 'threshold': 1.0, 'compare_mode': 'lazy', 'match_mode': 'identifier', 'result_mode': 'joker'}]) ] CFG_BIBMATCH_QUERY_TEMPLATES = { 'title': '[title]', 'title-author': '[title] [author]', 'reportnumber': 'reportnumber:[reportnumber]' } CFG_BIBMATCH_REMOTE_SLEEPTIME = 2.0 CFG_BIBMATCH_SEARCH_RESULT_MATCH_LIMIT = 15 CFG_BIBMATCH_MIN_VALIDATION_COMPARISONS = 2 CFG_BIBRANK_SELFCITES_PRECOMPUTE = 0 CFG_BIBRANK_SELFCITES_USE_BIBAUTHORID = 0 CFG_BIBRANK_SHOW_CITATION_GRAPHS = 1 CFG_BIBRANK_SHOW_CITATION_LINKS = 1 CFG_BIBRANK_SHOW_CITATION_STATS = 1 CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS = 1 CFG_BIBRANK_SHOW_DOWNLOAD_GRAPHS_CLIENT_IP_DISTRIBUTION = 0 CFG_BIBRANK_SHOW_DOWNLOAD_STATS = 1 CFG_BIBRANK_SHOW_READING_STATS = 1 CFG_BIBSCHED_EDITOR = which("vim") CFG_BIBSCHED_GC_TASKS_OLDER_THAN = 30 CFG_BIBSCHED_GC_TASKS_TO_ARCHIVE = ['bibupload', 'oairepositoryupdater', ] CFG_BIBSCHED_GC_TASKS_TO_REMOVE = [ 'bibindex', 'bibreformat', 'webcoll', 'bibrank', 'inveniogc', ] CFG_BIBSCHED_LOG_PAGER = which("less") CFG_BIBSCHED_LOGDIR = join(_cfg_prefix, "var", "log", "bibsched") CFG_BIBSCHED_MAX_ARCHIVED_ROWS_DISPLAY = 500 CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS = 1 CFG_BIBSCHED_NODE_TASKS = {} CFG_BIBSCHED_PROCESS_USER = "" CFG_BIBSCHED_REFRESHTIME = 5 CFG_BIBSCHED_TASKLET_PACKAGES = [ 'invenio.legacy.bibsched.tasklets', ] CFG_BIBSCHED_NON_CONCURRENT_TASKS = [] CFG_BIBSCHED_FLUSH_LOGS = 0 CFG_BIBSCHED_INCOMPATIBLE_TASKS = () CFG_BIBSCHED_NEVER_STOPS = 0 CFG_BIBSORT_BUCKETS = 1 CFG_BIBSORT_ENABLED = 1 CFG_BIBSORT_DEFAULT_FIELD = 'latest first' CFG_BIBSORT_DEFAULT_FIELD_ORDER = 'd' CFG_BIBUPLOAD_CONFLICTING_REVISION_TICKET_QUEUE = "" CFG_BIBUPLOAD_CONTROLLED_PROVENANCE_TAGS = ['6531_9', ] CFG_BIBUPLOAD_DELETE_FORMATS = ['hb', 'recjson'] CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS = 0 CFG_BIBUPLOAD_EXTERNAL_OAIID_PROVENANCE_TAG = "035__9" CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG = "035__a" CFG_BIBUPLOAD_EXTERNAL_SYSNO_TAG = "970__a" CFG_BIBUPLOAD_FFT_ALLOWED_EXTERNAL_URLS = [ ('http(s)?://.*', {}), ] CFG_BIBUPLOAD_FFT_ALLOWED_LOCAL_PATHS = ['/tmp', '/home', '/Users'] CFG_BIBUPLOAD_REFERENCE_TAG = "999" CFG_BIBUPLOAD_SERIALIZE_RECORD_STRUCTURE = 1 CFG_BIBUPLOAD_STRONG_TAGS = ['964', ] CFG_BIBUPLOAD_INTERNAL_DOI_PATTERN = "[^\w\W]" CFG_BIBUPLOAD_MATCH_DELETED_RECORDS = 1 CFG_BIBWORKFLOW_WORKER = "worker_celery" CFG_BROKER_URL = "amqp://guest@localhost:5672//" CFG_CELERY_RESULT_BACKEND = "amqp" CFG_CERN_SITE = 0 CFG_ORGANIZATION_IDENTIFIER = "" CFG_CROSSREF_EMAIL = "" CFG_CROSSREF_PASSWORD = "" CFG_CROSSREF_USERNAME = "" CFG_DEVEL_SITE = 0 CFG_DEVEL_TEST_DATABASE_ENGINES = {} CFG_DEVEL_TOOLS = [] CFG_EMAIL_BACKEND = "flask_email.backends.smtp.Mail" CFG_ERRORLIB_RESET_EXCEPTION_NOTIFICATION_COUNTER_AFTER = 14400 CFG_FLASK_DISABLED_BLUEPRINTS = [] CFG_HEPDATA_FIELD = "hepdataparent" CFG_HEPDATA_INDEX = "hepdataparent" CFG_HEPDATA_PLOTSIZE = 200 CFG_HEPDATA_THREADS_NUM = 1 CFG_HEPDATA_URL = "http://hepdata.cedar.ac.uk" CFG_ICON_CREATION_FORMAT_MAPPINGS = {'*': ['jpg']} CFG_INSPIRE_SITE = 0 CFG_INTBITSET_ENABLE_SANITY_CHECKS = False CFG_JSTESTDRIVER_PORT = 9876 CFG_MATHJAX_HOSTING = "local" CFG_MATHJAX_RENDERS_MATHML = True CFG_MISCUTIL_DEFAULT_PROCESS_TIMEOUT = 300 CFG_MISCUTIL_SMTP_HOST = "localhost" CFG_MISCUTIL_SMTP_PASS = "" CFG_MISCUTIL_SMTP_PORT = 25 CFG_MISCUTIL_SMTP_TLS = False CFG_MISCUTIL_SMTP_USER = "" CFG_MISCUTIL_SQL_RUN_SQL_MANY_LIMIT = 10000 CFG_MISCUTIL_SQL_USE_SQLALCHEMY = False CFG_OAI_DELETED_POLICY = "persistent" CFG_OAI_EXPIRE = 90000 CFG_OAI_FAILED_HARVESTING_EMAILS_ADMIN = True CFG_OAI_FAILED_HARVESTING_STOP_QUEUE = 1 CFG_OAI_FRIENDS = ['http://cds.cern.ch/oai2d', 'http://openaire.cern.ch/oai2d', 'http://export.arxiv.org/oai2', ] CFG_OAI_ID_FIELD = "909COo" CFG_OAI_ID_PREFIX = "atlantis.cern.ch" CFG_OAI_IDENTIFY_DESCRIPTION = """ http://localhost Free and unlimited use by anybody with obligation to refer to original \ record Full content, i.e. preprints may not be harvested by robots Submission restricted. Submitted documents are subject of approval by \ OAI repository admins. """ CFG_OAI_LICENSE_FIELD = "540__" CFG_OAI_LICENSE_PUBLISHER_SUBFIELD = "b" CFG_OAI_LICENSE_TERMS_SUBFIELD = "a" CFG_OAI_LICENSE_URI_SUBFIELD = "u" CFG_OAI_LOAD = 500 CFG_OAI_METADATA_FORMATS = { 'oai_dc': ('XOAIDC', 'http://www.openarchives.org/OAI/1.1/dc.xsd', 'http://purl.org/dc/elements/1.1/'), 'marcxml': ('XOAIMARC', 'http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd', 'http://www.loc.gov/MARC21/slim'), } CFG_OAI_PREVIOUS_SET_FIELD = "909COq" CFG_OAI_PROVENANCE_ALTERED_SUBFIELD = "t" CFG_OAI_PROVENANCE_BASEURL_SUBFIELD = "u" CFG_OAI_PROVENANCE_DATESTAMP_SUBFIELD = "d" CFG_OAI_PROVENANCE_HARVESTDATE_SUBFIELD = "h" CFG_OAI_PROVENANCE_METADATANAMESPACE_SUBFIELD = "m" CFG_OAI_PROVENANCE_ORIGINDESCRIPTION_SUBFIELD = "d" CFG_OAI_RIGHTS_CONTACT_SUBFIELD = "e" CFG_OAI_RIGHTS_DATE_SUBFIELD = "g" CFG_OAI_RIGHTS_FIELD = "542__" CFG_OAI_RIGHTS_HOLDER_SUBFIELD = "d" CFG_OAI_RIGHTS_STATEMENT_SUBFIELD = "f" CFG_OAI_RIGHTS_URI_SUBFIELD = "u" CFG_OAI_SAMPLE_IDENTIFIER = "oai:atlantis.cern.ch:123" CFG_OAI_SET_FIELD = "909COp" CFG_OAI_SLEEP = 2 CFG_OPENOFFICE_SERVER_HOST = "localhost" CFG_OPENOFFICE_SERVER_PORT = 2002 CFG_OPENOFFICE_USER = "nobody" CFG_PATH_ANY2DJVU = "" CFG_PATH_CONVERT = which("convert") CFG_PATH_DJVUPS = "" CFG_PATH_DJVUTXT = "" CFG_PATH_FFMPEG = "" CFG_PATH_FFPROBE = "" CFG_PATH_GFILE = which("file") CFG_PATH_GIT = which("git") CFG_PATH_GS = which("gs") CFG_PATH_GUNZIP = which("gunzip") CFG_PATH_GZIP = which("gzip") CFG_PATH_MD5SUM = "" CFG_PATH_MEDIAINFO = "" CFG_PATH_MYSQL = which("mysql") CFG_PATH_OCROSCRIPT = "" CFG_PATH_OPENOFFICE_PYTHON = which("python") CFG_PATH_PAMFILE = which("pdftoppm") CFG_PATH_PDF2PS = which("pdf2ps") CFG_PATH_PDFINFO = which("pdfinfo") CFG_PATH_PDFLATEX = which("pdflatex") CFG_PATH_PDFOPT = which("pdfopt") or which("cp") CFG_PATH_PDFTK = which("pdftk") CFG_PATH_PDFTOPPM = which("pdftoppm") CFG_PATH_PDFTOPS = which("pdftops") CFG_PATH_PDFTOTEXT = which("pdftotext") CFG_PATH_PHP = which("php") CFG_PATH_PS2PDF = which("ps2pdf") CFG_PATH_PSTOASCII = which("ps2ascii") CFG_PATH_PSTOTEXT = "" CFG_PATH_SVN = which("svn") CFG_PATH_TAR = which("tar") CFG_PATH_TIFF2PDF = which("tiff2pdf") CFG_PATH_WGET = which("wget") CFG_PLOTEXTRACTOR_CONTEXT_EXTRACT_LIMIT = 750 CFG_PLOTEXTRACTOR_CONTEXT_SENTENCE_LIMIT = 2 CFG_PLOTEXTRACTOR_CONTEXT_WORD_LIMIT = 75 CFG_PLOTEXTRACTOR_DISALLOWED_TEX = [ 'begin', 'end', 'section', 'includegraphics', 'caption', 'acknowledgements', ] CFG_PLOTEXTRACTOR_DOWNLOAD_TIMEOUT = 2.0 CFG_PLOTEXTRACTOR_SOURCE_BASE_URL = "http://arxiv.org/" CFG_PLOTEXTRACTOR_SOURCE_PDF_FOLDER = "pdf/" CFG_PLOTEXTRACTOR_SOURCE_TARBALL_FOLDER = "e-print/" CFG_REDIS_HOSTS = {'default': [{'db': 0, 'host': '127.0.0.1', 'port': 6379}]} CFG_REFEXTRACT_KBS_OVERRIDE = {} CFG_REFEXTRACT_TICKET_QUEUE = None CFG_SCOAP3_SITE = 0 CFG_SITE_ADMIN_EMAIL = "info@invenio-software.org" CFG_SITE_ADMIN_EMAIL_EXCEPTIONS = 1 CFG_SITE_EMERGENCY_EMAIL_ADDRESSES = {} CFG_SITE_LANG = "en" CFG_SITE_LANGS = ['af', 'ar', 'bg', 'ca', 'cs', 'de', 'el', 'en', 'es', 'fr', 'hr', 'gl', 'ka', 'it', 'rw', 'lt', 'hu', 'ja', 'no', 'pl', 'pt', 'ro', 'ru', 'sk', 'sv', 'uk', 'zh_CN', 'zh_TW', ] CFG_SITE_NAME = "Atlantis Institute of Fictive Science" CFG_SITE_RECORD = "record" SECRET_KEY = "change_me" CFG_SITE_SECURE_URL = "http://localhost:4000" CFG_SITE_SUPPORT_EMAIL = "info@invenio-software.org" CFG_SITE_URL = "http://localhost:4000" CFG_SOLR_URL = "" CFG_VERSION = __version__ CFG_WEB_API_KEY_ALLOWED_URL = [] CFG_WEBALERT_ALERT_ENGINE_EMAIL = "info@invenio-software.org" CFG_WEBALERT_MAX_NUM_OF_CHARS_PER_LINE_IN_ALERT_EMAIL = 72 CFG_WEBALERT_MAX_NUM_OF_RECORDS_IN_ALERT_EMAIL = 20 CFG_WEBALERT_SEND_EMAIL_NUMBER_OF_TRIES = 3 CFG_WEBALERT_SEND_EMAIL_SLEEPTIME_BETWEEN_TRIES = 300 CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_BIBSCHED = 5 CFG_WEBAUTHORPROFILE_CACHE_EXPIRED_DELAY_LIVE = 7 CFG_WEBAUTHORPROFILE_MAX_AFF_LIST = 100 CFG_WEBAUTHORPROFILE_MAX_COAUTHOR_LIST = 100 CFG_WEBAUTHORPROFILE_MAX_COLLAB_LIST = 100 CFG_WEBAUTHORPROFILE_MAX_HEP_CHOICES = 10 CFG_WEBAUTHORPROFILE_MAX_KEYWORD_LIST = 100 CFG_WEBAUTHORPROFILE_USE_BIBAUTHORID = False CFG_WEBAUTHORPROFILE_ALLOWED_FIELDCODES = [ 'Astrophysics', 'Accelerators', 'Computing', 'Experiment-HEP', 'Gravitation and Cosmology', 'Instrumentation', 'Lattice', 'Math and Math Physics', 'Theory-Nucl', 'Other', 'Phenomenology-HEP', 'General Physics', 'Theory-HEP', 'Experiment-Nucl' ] CFG_WEBAUTHORPROFILE_CFG_HEPNAMES_EMAIL = "authors@inspirehep.net" CFG_WEBAUTHORPROFILE_MAX_FIELDCODE_LIST = 100 CFG_WEBAUTHORPROFILE_ORCID_ENDPOINT_PUBLIC = "http://pub.orcid.org/" CFG_WEBAUTHORPROFILE_ORCID_ENDPOINT_MEMBER = "http://api.orcid.org/" CFG_WEBAUTHORPROFILE_USE_ALLOWED_FIELDCODES = True CFG_WEBBASKET_MAX_NUMBER_OF_DISPLAYED_BASKETS = 20 CFG_WEBBASKET_USE_RICH_TEXT_EDITOR = False CFG_WEBCOMMENT_ADMIN_NOTIFICATION_LEVEL = 1 CFG_WEBCOMMENT_ALERT_ENGINE_EMAIL = "info@invenio-software.org" CFG_WEBCOMMENT_ALLOW_COMMENTS = 1 CFG_WEBCOMMENT_ALLOW_REVIEWS = 1 CFG_WEBCOMMENT_ALLOW_SHORT_REVIEWS = 0 CFG_WEBCOMMENT_AUTHOR_DELETE_COMMENT_OPTION = 1 CFG_WEBCOMMENT_DEFAULT_MODERATOR = "info@invenio-software.org" CFG_WEBCOMMENT_EMAIL_REPLIES_TO = { 'Articles': ['506__d', '506__m'], } CFG_WEBCOMMENT_MAX_ATTACHED_FILES = 5 CFG_WEBCOMMENT_MAX_ATTACHMENT_SIZE = 5242880 CFG_WEBCOMMENT_MAX_COMMENT_THREAD_DEPTH = 1 CFG_WEBCOMMENT_NB_COMMENTS_IN_DETAILED_VIEW = 1 CFG_WEBCOMMENT_NB_REPORTS_BEFORE_SEND_EMAIL_TO_ADMIN = 5 CFG_WEBCOMMENT_NB_REVIEWS_IN_DETAILED_VIEW = 1 CFG_WEBCOMMENT_RESTRICTION_DATAFIELD = { 'Articles': '5061_a', 'Pictures': '5061_a', 'Theses': '5061_a', } CFG_WEBCOMMENT_ROUND_DATAFIELD = { 'Articles': '562__c', 'Pictures': '562__c', } CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_COMMENTS_IN_SECONDS = 20 CFG_WEBCOMMENT_TIMELIMIT_PROCESSING_REVIEWS_IN_SECONDS = 20 CFG_WEBCOMMENT_USE_MATHJAX_IN_COMMENTS = 1 CFG_WEBCOMMENT_USE_RICH_TEXT_EDITOR = False CFG_WEBDEPOSIT_UPLOAD_FOLDER = "var/tmp/webdeposit_uploads" CFG_WEBLINKBACK_TRACKBACK_ENABLED = 0 CFG_WEBMESSAGE_DAYS_BEFORE_DELETE_ORPHANS = 60 CFG_WEBMESSAGE_MAX_NB_OF_MESSAGES = 30 CFG_WEBMESSAGE_MAX_SIZE_OF_MESSAGE = 20000 CFG_WEBSEARCH_ADVANCEDSEARCH_PATTERN_BOX_WIDTH = 30 CFG_WEBSEARCH_AUTHOR_ET_AL_THRESHOLD = 3 CFG_WEBSEARCH_CALL_BIBFORMAT = 0 CFG_WEBSEARCH_CITESUMMARY_SELFCITES_THRESHOLD = 2000 CFG_WEBSEARCH_CITESUMMARY_SCAN_THRESHOLD = 20000 CFG_WEBSEARCH_CREATE_SIMILARLY_NAMED_AUTHORS_LINK_BOX = 1 CFG_WEBSEARCH_DEF_RECORDS_IN_GROUPS = 10 CFG_WEBSEARCH_DETAILED_META_FORMAT = "hdm" CFG_WEBSEARCH_DISPLAY_NEAREST_TERMS = 1 CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR = True CFG_WEBSEARCH_ENABLE_OPENGRAPH = False CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_MAXRESULTS = 10 CFG_WEBSEARCH_EXTERNAL_COLLECTION_SEARCH_TIMEOUT = 5 CFG_WEBSEARCH_FIELDS_CONVERT = {} CFG_WEBSEARCH_FULLTEXT_SNIPPETS = { '': 4, } CFG_WEBSEARCH_FULLTEXT_SNIPPETS_CHARS = { '': 100, } CFG_WEBSEARCH_FULLTEXT_SNIPPETS_GENERATOR = "native" CFG_WEBSEARCH_I18N_LATEST_ADDITIONS = 0 CFG_WEBSEARCH_INSTANT_BROWSE = 10 CFG_WEBSEARCH_INSTANT_BROWSE_RSS = 25 CFG_WEBSEARCH_LIGHTSEARCH_PATTERN_BOX_WIDTH = 60 CFG_WEBSEARCH_MAX_RECORDS_IN_GROUPS = 200 CFG_WEBSEARCH_NARROW_SEARCH_SHOW_GRANDSONS = 1 CFG_WEBSEARCH_NB_RECORDS_TO_SORT = 1000 CFG_WEBSEARCH_PREV_NEXT_HIT_FOR_GUESTS = 1 CFG_WEBSEARCH_PREV_NEXT_HIT_LIMIT = 1000 CFG_WEBSEARCH_RSS_I18N_COLLECTIONS = [] CFG_WEBSEARCH_RSS_MAX_CACHED_REQUESTS = 1000 CFG_WEBSEARCH_RSS_TTL = 360 CFG_WEBSEARCH_SEARCH_CACHE_SIZE = 1 CFG_WEBSEARCH_SEARCH_CACHE_TIMEOUT = 600 CFG_WEBSEARCH_SHOW_COMMENT_COUNT = 1 CFG_WEBSEARCH_SHOW_REVIEW_COUNT = 1 CFG_WEBSEARCH_SIMPLESEARCH_PATTERN_BOX_WIDTH = 40 CFG_WEBSEARCH_SPIRES_SYNTAX = 1 CFG_WEBSEARCH_SPLIT_BY_COLLECTION = 1 CFG_WEBSEARCH_SYNONYM_KBRS = { 'journal': ['SEARCH-SYNONYM-JOURNAL', 'leading_to_number'], } CFG_WEBSEARCH_USE_ALEPH_SYSNOS = 0 CFG_WEBSEARCH_USE_MATHJAX_FOR_FORMATS = [] CFG_WEBSEARCH_VIEWRESTRCOLL_POLICY = "ANY" CFG_WEBSEARCH_WILDCARD_LIMIT = 50000 CFG_WEBSESSION_ADDRESS_ACTIVATION_EXPIRE_IN_DAYS = 3 CFG_WEBSESSION_EXPIRY_LIMIT_DEFAULT = 2 CFG_WEBSESSION_EXPIRY_LIMIT_REMEMBER = 365 CFG_WEBSESSION_IPADDR_CHECK_SKIP_BITS = 0 CFG_WEBSESSION_NOT_CONFIRMED_EMAIL_ADDRESS_EXPIRE_IN_DAYS = 10 CFG_WEBSESSION_RESET_PASSWORD_EXPIRE_IN_DAYS = 3 CFG_WEBSESSION_STORAGE = "redis" CFG_WEBSTAT_BIBCIRCULATION_START_YEAR = "" CFG_WEBSTYLE_CDSPAGEBOXLEFTBOTTOM = "" CFG_WEBSTYLE_CDSPAGEBOXLEFTTOP = "" CFG_WEBSTYLE_CDSPAGEBOXRIGHTBOTTOM = "" CFG_WEBSTYLE_CDSPAGEBOXRIGHTTOP = "" CFG_WEBSTYLE_EMAIL_ADDRESSES_OBFUSCATION_MODE = 2 CFG_WEBSTYLE_HTTP_STATUS_ALERT_LIST = ['404r', '400', '5*', '41*', ] CFG_WEBSTYLE_HTTP_USE_COMPRESSION = 0 CFG_WEBSTYLE_REVERSE_PROXY_IPS = [] CFG_WEBSTYLE_TEMPLATE_SKIN = "default" CFG_WEBSUBMIT_USE_MATHJAX = 0 CFG_XAPIAN_ENABLED = "" CFG_WEBSEARCH_DEFAULT_SEARCH_INTERFACE = 0 CFG_WEBSEARCH_ENABLED_SEARCH_INTERFACES = [0, 1, 2] # END OF GENERATED FILE diff --git a/invenio/base/scripts/apache.py b/invenio/base/scripts/apache.py index 2dc34b2ab..9affb3b55 100644 --- a/invenio/base/scripts/apache.py +++ b/invenio/base/scripts/apache.py @@ -1,254 +1,254 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. from __future__ import print_function import os import socket from invenio.ext.script import Manager, change_command_name manager = Manager(usage="Perform Apache operations.") def _detect_ip_address(): """Detect IP address of this computer. Useful for creating Apache vhost conf snippet on RHEL like machines. However, if wanted site is 0.0.0.0, then use that, since we are running inside Docker. :return: IP address, or '*' if cannot detect :rtype: string .. note:: creates socket for real in order to detect real IP address, not the loopback one. """ from invenio.base.globals import cfg if '0.0.0.0' in cfg.get('CFG_SITE_URL'): return '0.0.0.0' try: s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(('invenio-software.org', 0)) return s.getsockname()[0] except Exception: return '*' def _grep_version_from_executable(path_to_exec, version_regexp): """Try to detect a program version. Grep in its binary PATH_TO_EXEC and looking for VERSION_REGEXP. Return program version as a string. Return empty string if not succeeded. """ from invenio.utils.shell import run_shell_command exec_version = "" if os.path.exists(path_to_exec): dummy1, cmd2_out, dummy2 = run_shell_command( "strings %s | grep %s", (path_to_exec, version_regexp)) if cmd2_out: for cmd2_out_line in cmd2_out.split("\n"): if len(cmd2_out_line) > len(exec_version): # the longest the better exec_version = cmd2_out_line return exec_version @manager.command def version(separator='\n', formatting='{version} [{executable}]'): """ Try to detect Apache version by localizing httpd or apache executables and grepping inside binaries. Return list of all found Apache versions and paths. (For a given executable, the returned format is 'apache_version [apache_path]'.) Return empty list if no success. """ from invenio.utils.shell import run_shell_command out = [] dummy1, cmd_out, dummy2 = run_shell_command("locate bin/httpd bin/apache") for apache in cmd_out.split("\n"): apache_version = _grep_version_from_executable(apache, '^Apache\/') if apache_version: out.append(formatting.format(version=apache_version, executable=apache)) if separator is None: return out return separator.join(out) @manager.option('-f', '--force', dest='force', action='store_true') @manager.option('--no-ssl', dest='no_ssl', action='store_true') @change_command_name def create_config(force=False, no_ssl=False): """ Create Apache configuration files for this site, keeping previous files in a backup copy. """ import os import pwd import pkg_resources import sys import shutil from flask import current_app from jinja2 import TemplateNotFound from invenio.ext.template import render_template_to_string from invenio.utils.text import wrap_text_in_a_box - CFG_PREFIX = current_app.config.get('CFG_PREFIX', '') + CFG_ETCDIR = current_app.config.get('CFG_ETCDIR', '') def get_context(): conf = current_app.config ## Apache vhost conf file is distro specific, so analyze needs: # Gentoo (and generic defaults): listen_directive_needed = True ssl_pem_directive_needed = False - ssl_pem_path = CFG_PREFIX + '/etc/apache/ssl/apache.pem' - ssl_crt_path = CFG_PREFIX + '/etc/apache/ssl/server.crt' - ssl_key_path = CFG_PREFIX + '/etc/apache/ssl/server.key' + ssl_pem_path = CFG_ETCDIR + '/apache/ssl/apache.pem' + ssl_crt_path = CFG_ETCDIR + '/apache/ssl/server.crt' + ssl_key_path = CFG_ETCDIR + '/apache/ssl/server.key' vhost_ip_address_needed = False wsgi_socket_directive_needed = False # Debian: if os.path.exists(os.path.sep + 'etc' + os.path.sep + 'debian_version'): listen_directive_needed = False ssl_pem_directive_needed = True ssl_pem_path = '/etc/apache2/ssl/apache.pem' ssl_crt_path = '/etc/apache2/ssl/server.crt' ssl_key_path = '/etc/apache2/ssl/server.key' # RHEL/SLC: if os.path.exists(os.path.sep + 'etc' + os.path.sep + 'redhat-release'): listen_directive_needed = False ssl_crt_path = '/etc/pki/tls/certs/localhost.crt' ssl_key_path = '/etc/pki/tls/private/localhost.key' vhost_ip_address_needed = True wsgi_socket_directive_needed = True # maybe we are using non-standard ports? vhost_site_url = conf.get('CFG_SITE_URL').replace("http://", "") if vhost_site_url.startswith("https://"): ## The installation is configured to require HTTPS for any connection vhost_site_url = vhost_site_url.replace("https://", "") vhost_site_url_port = '80' vhost_site_secure_url = conf.get('CFG_SITE_SECURE_URL').replace("https://", "").replace("http://", "") vhost_site_secure_url_port = '443' if ':' in vhost_site_url: vhost_site_url, vhost_site_url_port = vhost_site_url.split(':', 1) if ':' in vhost_site_secure_url: vhost_site_secure_url, vhost_site_secure_url_port = vhost_site_secure_url.split(':', 1) if vhost_site_url_port != '80' or vhost_site_secure_url_port != '443': listen_directive_needed = True static_root = current_app.config['COLLECT_STATIC_ROOT'] if not os.path.exists(static_root): os.mkdir(static_root) def prepare_alias(filename): if os.path.isdir(os.path.join(static_root, filename)): return '/%s/' % (filename, ) return '/%s' % (filename, ) aliases = map(prepare_alias, os.listdir(static_root)) apc1 = {'vhost_site_url_port': vhost_site_url_port, 'servername': vhost_site_url, 'serveralias': vhost_site_url.split('.')[0], 'vhost_ip_address': vhost_ip_address_needed and _detect_ip_address() or '*', 'wsgi_socket_directive_needed': wsgi_socket_directive_needed, 'listen_directive_needed': listen_directive_needed, 'aliases': aliases, } apc2 = {'vhost_site_url_port': vhost_site_secure_url_port, 'servername': vhost_site_secure_url, 'serveralias': vhost_site_secure_url.split('.')[0], 'vhost_ip_address': vhost_ip_address_needed and _detect_ip_address() or '*', 'wsgi_socket_directive_needed': wsgi_socket_directive_needed, 'ssl_pem_directive': ssl_pem_directive_needed and 'SSLCertificateFile %s' % ssl_pem_path or '#SSLCertificateFile %s' % ssl_pem_path, 'ssl_crt_directive': ssl_pem_directive_needed and '#SSLCertificateFile %s' % ssl_crt_path or 'SSLCertificateFile %s' % ssl_crt_path, 'ssl_key_directive': ssl_pem_directive_needed and '#SSLCertificateKeyFile %s' % ssl_key_path or 'SSLCertificateKeyFile %s' % ssl_key_path, 'listen_directive_needed': listen_directive_needed, 'aliases': aliases, } return [apc1, apc2] current_app.config.update( SYS_PREFIX=sys.prefix, CFG_RUNNING_AS_USER=pwd.getpwuid(os.getuid())[0], CFG_WSGIDIR=os.path.abspath( pkg_resources.resource_filename('invenio', ''))) apache_conf_dir = current_app.instance_path + os.sep + 'apache' print(">>> Going to create Apache conf files...") conf_files = ['invenio-apache-vhost.conf', 'invenio-apache-vhost-ssl.conf'] conf_files = conf_files[:1 if no_ssl else 2] if not os.path.exists(apache_conf_dir): os.mkdir(apache_conf_dir) for local_file, context in zip(conf_files, get_context()[:1 if no_ssl else 2]): print(">>> Writing %s ..." % local_file) try: apache_vhost_file = apache_conf_dir + os.sep + local_file if os.path.exists(apache_vhost_file): shutil.copy(apache_vhost_file, apache_vhost_file + '.OLD') with open(apache_vhost_file, 'w') as f: out = render_template_to_string(local_file + '.tpl', os=os, **context) print(out, file=f) except TemplateNotFound: print("Could not find template %s".format(local_file), file=sys.stderr) print(wrap_text_in_a_box("""\ Apache virtual host configuration file(s) for your Invenio site was(were) created. Please check created file(s) and activate virtual host(s). For example, you can put the following include statements in your httpd.conf:\n %s Please see the INSTALL file for more details. """ % '\n\n'.join(tuple(map( lambda x: "Include " + apache_conf_dir.encode('utf-8') + os.sep + x, list(conf_files[:1 if no_ssl else 2])))))) print(">>> Apache conf files created.") def main(): from invenio.base.factory import create_app app = create_app() manager.app = app manager.run() if __name__ == '__main__': main() diff --git a/invenio/base/scripts/demosite.py b/invenio/base/scripts/demosite.py index 42309b09a..c06d5aa3b 100644 --- a/invenio/base/scripts/demosite.py +++ b/invenio/base/scripts/demosite.py @@ -1,153 +1,150 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Perform demosite operations.""" from __future__ import print_function import os import sys import warnings from itertools import count import pkg_resources from invenio.base.utils import run_py_func from invenio.ext.script import Manager warnings.warn("Use of `inveniomanage demosite populate` is being deprecated. " "Please use `uploader` module to insert demo records.", PendingDeprecationWarning) manager = Manager(usage=__doc__) # Shortcuts for manager options to keep code DRY. option_yes_i_know = manager.option('--yes-i-know', action='store_true', dest='yes_i_know', help='use with care!') option_default_data = manager.option('--no-data', action='store_false', dest='default_data', help='do not populate tables with ' 'default data') option_file = manager.option('-f', '--file', dest='files', action='append', help='data file to use') option_jobid = manager.option('-j', '--job-id', dest='job_id', type=int, default=0, help='bibsched starting job id') option_extrainfo = manager.option('-e', '--extra-info', dest='extra_info', action='append', help='extraneous parameters') option_packages = manager.option('-p', '--packages', dest='packages', action='append', default=[], help='package import name (repeteable)') @option_packages @option_default_data @option_file @option_jobid @option_extrainfo @option_yes_i_know def populate(packages=[], default_data=True, files=None, job_id=0, extra_info=None, yes_i_know=False): """Load demo records. Useful for testing purposes.""" from invenio.utils.text import wrap_text_in_a_box, wait_for_user # Load cli interfaces for tools that we are going to need from invenio.legacy.bibupload.engine import main as bibupload from invenio.legacy.bibindex.engine import main as bibindex - from invenio.legacy.bibformat.bibreformat import main as bibreformat from invenio.legacy.oairepository.updater import main as oairepositoryupdater from invenio.legacy.bibsort.daemon import main as bibsort from invenio.legacy.bibdocfile.cli import main as bibdocfile from invenio.legacy.bibrank.cli import main as bibrank ## Step 0: confirm deletion wait_for_user(wrap_text_in_a_box( "WARNING: You are going to override data in tables!" )) if not default_data: print('>>> Default data has been skiped (--no-data).') return if not packages: packages = ['invenio_demosite.base'] from werkzeug.utils import import_string map(import_string, packages) from invenio.ext.sqlalchemy import db print(">>> Going to load demo records...") db.session.execute("TRUNCATE schTASK") db.session.commit() if files is None: files = [pkg_resources.resource_filename( 'invenio', os.path.join('testsuite', 'data', 'demo_record_marc_data.xml'))] # upload demo site files: bibupload_flags = '-i' if extra_info is not None and 'force-recids' in extra_info: bibupload_flags = '-i -r --force' for f in files: job_id += 1 for cmd in ( (bibupload, "bibupload -u admin %s %s" % (bibupload_flags, f)), (bibupload, "bibupload %d" % (job_id)) ): if run_py_func(*cmd, passthrough=True).exit_code: print("ERROR: failed execution of", *cmd) sys.exit(1) i = count(job_id + 1).next for cmd in ( (bibdocfile, "bibdocfile --textify --with-ocr --recid 97"), (bibdocfile, "bibdocfile --textify --all"), (bibindex, "bibindex -u admin"), (bibindex, "bibindex %d" % i()), (bibindex, "bibindex -u admin -w global"), (bibindex, "bibindex %d" % i()), - (bibreformat, "bibreformat -u admin -o HB"), - (bibreformat, "bibreformat %d" % i()), (bibrank, "bibrank -u admin"), (bibrank, "bibrank %d" % i()), (bibsort, "bibsort -u admin -R"), (bibsort, "bibsort %d" % i()), (oairepositoryupdater, "oairepositoryupdater -u admin"), (oairepositoryupdater, "oairepositoryupdater %d" % i()), (bibupload, "bibupload %d" % i()), ): if run_py_func(*cmd, passthrough=True).exit_code: print("ERROR: failed execution of", *cmd) sys.exit(1) print(">>> Demo records loaded successfully.") def main(): """Start the commandline manager.""" from invenio.base.factory import create_app app = create_app() manager.app = app manager.run() if __name__ == '__main__': main() diff --git a/invenio/base/templates/invenio-apache-vhost.conf.tpl b/invenio/base/templates/invenio-apache-vhost.conf.tpl index c849caf3d..58860c80d 100644 --- a/invenio/base/templates/invenio-apache-vhost.conf.tpl +++ b/invenio/base/templates/invenio-apache-vhost.conf.tpl @@ -1,42 +1,42 @@ {# # This file is part of Invenio. -# Copyright (C) 2013 CERN. +# Copyright (C) 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. #} {%- extends "invenio-apache-vhost.tpl" -%} {%- block header -%} AddDefaultCharset UTF-8 ServerSignature Off ServerTokens Prod NameVirtualHost {{ vhost_ip_address }}:{{ vhost_site_url_port }} {{ '#' if not listen_directive_needed }}{{ 'Listen ' + vhost_site_url_port}} -{{ '#' if not wsgi_socket_directive_needed }}WSGISocketPrefix {{ [config.CFG_PREFIX, 'var', 'run']|path_join }} +{{ '#' if not wsgi_socket_directive_needed }}WSGISocketPrefix {{ config.CFG_RUNDIR }} {{ super() }} {%- endblock header -%} {%- set wsgi_processes = config.get('APACHE_WSGI_DAEMON_PROCESSES', 5) -%} {%- if config.DEBUG -%} {%- set wsgi_processes = 1 -%} {%- endif -%} {%- block wsgi %} WSGIDaemonProcess invenio processes={{ wsgi_processes }} threads=1 user={{ config.CFG_RUNNING_AS_USER }} display-name=%{GROUP} inactivity-timeout=3600 maximum-requests=10000 WSGIImportScript {{ config.CFG_WSGIDIR }}/invenio.wsgi process-group=invenio application-group=%{GLOBAL} {{ super() }} {%- endblock wsgi -%} {%- block auth_shibboleth -%} {# shibboleth is allowed only on https #} {%- endblock auth_shibboleth -%} diff --git a/invenio/base/templates/invenio-apache-vhost.tpl b/invenio/base/templates/invenio-apache-vhost.tpl index ccf14afb2..cef76013b 100644 --- a/invenio/base/templates/invenio-apache-vhost.tpl +++ b/invenio/base/templates/invenio-apache-vhost.tpl @@ -1,184 +1,184 @@ {# # This file is part of Invenio. # Copyright (C) 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. #} {# Apache configuration template ============================= blocks: ------- - configuration - header - virtual_host - server - directory_web - logging - alliases - wsgi - xsendfile_directive - directory_wsgi - deflate_directive - auth_shibboleth variables: ---------- - log_suffix: adds string to apache log names `{% set log_suffix = '' %}` (default: '') #} {%- block configuration -%} {%- block header -%} WSGIRestrictStdout Off {%- set pythonhome = config.get('APACHE_PYTHON_HOME', config.SYS_PREFIX) %} {%- if pythonhome %} WSGIPythonHome {{pythonhome}} {%- endif %} deny from all deny from all {%- endblock header -%} {%- block virtual_host %} {%- block server %} ServerName {{ servername }} ServerAlias {{ config.get('APACHE_SERVER_ALIASES', serveralias) }} ServerAdmin {{ config.CFG_SITE_ADMIN_EMAIL }} {%- endblock server -%} {%- block directory_web %} DocumentRoot {{ config.COLLECT_STATIC_ROOT }} DirectorySlash Off Options +FollowSymLinks +MultiViews -Indexes AllowOverride None = 2.4> Require all granted Order allow,deny Allow from all {%- endblock directory_web -%} {%- block logging %} ErrorLog {{ config.CFG_LOGDIR }}/apache{{ log_suffix }}.err LogLevel warn LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" combined_with_timing CustomLog {{ config.CFG_LOGDIR }}/apache{{ log_suffix }}.log combined_with_timing {%- endblock logging -%} {%- block aliases %} #FIXME DirectoryIndex index.en.html index.html AliasMatch /sitemap-(.*) {{ config.CFG_WEBDIR }}/sitemap-$1 {%- endblock aliases -%} {%- block wsgi %} # Name of the WSGI entry point. # Change it at the *three* locations if you have to. {%- set script_alias = "/wsgi" %} SetEnv WSGI_SCRIPT_ALIAS {{ script_alias }} WSGIScriptAlias {{ script_alias }} {{ config.CFG_WSGIDIR }}/invenio.wsgi WSGIPassAuthorization On RewriteEngine on RewriteCond {{ config.COLLECT_STATIC_ROOT }}%{REQUEST_FILENAME} !-f RewriteRule ^(.*)$ {{ script_alias }}$1 [PT,L] {% endblock wsgi -%} {%- block xsendfile_directive %} {{ '#' if not config.CFG_BIBDOCFILE_USE_XSENDFILE }}XSendFile On {%- for xsfp in [config.CFG_BIBDOCFILE_FILEDIR, config.CFG_WEBDIR, config.CFG_WEBSUBMIT_STORAGEDIR, config.DEPOSIT_STORAGEDIR, config.CFG_TMPDIR, - [config.CFG_PREFIX, 'var', 'tmp', 'attachfile']|path_join, - [config.CFG_PREFIX, 'var', 'data', 'comments']|path_join, - [config.CFG_PREFIX, 'var', 'data', 'baskets', 'comments']|path_join, + [config.CFG_TMPDIR, 'attachfile']|path_join, + [config.CFG_DATADIR, 'comments']|path_join, + [config.CFG_DATADIR, 'baskets', 'comments']|path_join, '/tmp'] %} {% if xsfp %}{{ '#' if not config.CFG_BIBDOCFILE_USE_XSENDFILE }}XSendFilePath {{ xsfp }}{% endif %} {%- endfor -%} {%- endblock xsendfile_directive -%} {%- block directory_wsgi %} WSGIProcessGroup invenio WSGIApplicationGroup %{GLOBAL} Options +FollowSymLinks +MultiViews AllowOverride None = 2.4> Require all granted Order allow,deny Allow from all {%- endblock directory_wsgi -%} {%- block deflate_directive %} {% if config.CFG_WEBSTYLE_HTTP_USE_COMPRESSION %} ## Configuration snippet taken from: ## SetOutputFilter DEFLATE # Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html # Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip # MSIE masquerades as Netscape, but it is fine # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48 # the above regex won't work. You can use the following # workaround to get the desired effect: BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html # Don't compress images SetEnvIfNoCase Request_URI \ \.(?:gif|jpe?g|png)$ no-gzip dont-vary # Make sure proxies don't deliver the wrong content Header append Vary User-Agent env=!dont-vary {% endif -%} {%- endblock deflate_directive -%} {%- block etags -%} # Don't do etags for files since in a load balanced environment, two # servers will compute different etags for the same component. FileETag None {%- endblock etags -%} {%- block auth_shibboleth -%} {%- if config.CFG_EXTERNAL_AUTH_USING_SSO %} SSLRequireSSL # The modules only work using HTTPS AuthType shibboleth ShibRequireSession On ShibRequireAll On ShibExportAssertion Off require valid-user {% endif -%} {%- endblock auth_shibboleth %} {%- endblock virtual_host -%} {%- endblock -%} diff --git a/invenio/ext/jinja2hacks.py b/invenio/ext/jinja2hacks.py deleted file mode 100644 index 401549fe4..000000000 --- a/invenio/ext/jinja2hacks.py +++ /dev/null @@ -1,141 +0,0 @@ -# -*- coding: utf-8 -*- -# This file is part of Invenio. -# Copyright (C) 2012, 2013, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""Support passing legacy Invenio str objects to Jinja2 templates.""" - -import warnings - -try: - from markupsafe import Markup as jinja2_Markup, escape as jinja2_escape -except ImportError: - from jinja2._markupsafe import Markup as jinja2_Markup, \ - escape as jinja2_escape - -from invenio.utils.deprecation import RemovedInInvenio22Warning - -warnings.warn( - "Jinja2Hacks will be disabled in 2.1 and removed in 2.2. " - "Please convert all strings in Jinja2 templates to unicode.", - RemovedInInvenio22Warning -) - - -def setup_app(app): - """Jinja2 require all strings to be unicode objects. - - Invenio however operates with UTF8 encoded str objects. Jinja2 will - automatically convert non-unicode objects into unicode objects, but via the - ascii codec. This function replaces the escape function and Markup class in - Jinja2/MarkupSafe, to use the utf8 codec when converting 8-bit str objects - into unicode objects. - - Ideally Jinja2/MarkupSafe should allow specifying which default encoding to - use when decoding strings. Other alternatives is to decode any str object - into unicode prior to passing the values to Jinja2 methods. This will - however require large changes over the entire Invenio codebase, with the - risk of introducing many errors. This runtime hack is unfortunately - currently the least intrusive way to fix the str to unicode decoding. - """ - # Jinja2 will try to load escape method and Markup class from a variety of - # different modules. First it will try from MarkupSafe package, then from - # jinja2._markupsafe._speedup, then jinja2._markupsafe._native. Ideally, we - # should only replace the function and class at the implementing module. - # However, due to Python's package/module loading behaviour, the function - # and class will be imported into other jinja2 modules as soon as we try to - # import the module implementing the function and class. Hence, we need to - # replace the function and class in the modules where it has already been - # imported. - import jinja2 - import jinja2.runtime - import jinja2.utils - import jinja2.nodes - import jinja2.filters - import jinja2.ext - import jinja2.environment - import jinja2.compiler - - # Escape function replacement in Jinja2 library - try: - jinja2._markupsafe.escape = utf8escape - except AttributeError: - pass - jinja2.runtime.escape = utf8escape - jinja2.utils.escape = utf8escape - jinja2.filters.escape = utf8escape - jinja2.compiler.escape = utf8escape - jinja2.escape = utf8escape - - # Markup class replacement in Jinja2 library - try: - jinja2._markupsafe.Markup = Markup - except AttributeError: - pass - jinja2.runtime.Markup = Markup - jinja2.utils.Markup = Markup - jinja2.filters.Markup = Markup - jinja2.compiler.Markup = Markup - jinja2.Markup = Markup - jinja2.nodes.Markup = Markup - jinja2.ext.Markup = Markup - jinja2.environment.Markup = Markup - - # Escape/Markup replacement in MarkupSafe library. - # FIXME causes recursive calls in `Markup.__new__` and `escape` - # try: - # import markupsafe - # markupsafe.escape = utf8escape - # #markupsafe.Markup = Markup - # except ImportError: - # pass - - return app - - -def utf8escape(s): - """UTF8-8-bit-string-friendly replacement for MarkupSafe escape function. - - WARNING: Do not use this method. Use jinja2.escape() instead. - """ - if isinstance(s, str): - warnings.warn("Convert string '{0}' in template to unicode.".format(s), - DeprecationWarning, stacklevel=3) - return jinja2_escape(s.decode('utf8')) - return jinja2_escape(s) -# Ensure function name is identical to replaced function. -utf8escape.__name__ = jinja2_escape.__name__ - - -class Markup(jinja2_Markup): - - """Markup replacement class. - - Forces the use of utf8 codec for decoding 8-bit strings, in case no - encoding is specified. - - WARNING: Do not use this class. Use jinja2.Markup instead. - """ - - def __new__(cls, base=u'', encoding=None, errors='strict'): - """Add encoding for base of type str.""" - if encoding is None and isinstance(base, str): - encoding = 'utf8' - warnings.warn( - "Convert string '{0}' in template to unicode.".format(base), - DeprecationWarning, stacklevel=3) - return jinja2_Markup.__new__(cls, base=base, encoding=encoding, - errors=errors) diff --git a/invenio/legacy/bibclassify/config.py b/invenio/legacy/bibclassify/config.py index 1c51939f7..fa3589f58 100644 --- a/invenio/legacy/bibclassify/config.py +++ b/invenio/legacy/bibclassify/config.py @@ -1,319 +1,318 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013, 2014 CERN. +# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """ BibClassify configuration file. When writing changes, please either delete the cached ontology in your temporary directory or use the rebuild-cache option in order to regenerate the cached ontology. If you want to change this configuration, we recommend to create a local configuration file names 'bibclassify_config_local.py' that contains the changes to apply. """ from __future__ import unicode_literals import re import logging import sys import os from invenio import config VERSION = '0.4.9' logging_level = logging.ERROR # ------------- main config ----------- # Save generated kw into the database? # daemon does that CFG_DB_SAVE_KW = True # Number of keywords that are printed by default (this limits single keywords, # composite keywords, and acronyms - not author keywords) CFG_BIBCLASSIFY_DEFAULT_OUTPUT_NUMBER = 20 # The main marc xml field where to find/save the keywords, including the # indicators CFG_MAIN_FIELD = '6531_' # Other fields to take from the marc xml when generating tagcloud/list of # keywords. CFG_OTHER_FIELDS = ['6950_'] # Where to save author supplied keywords CFG_AUTH_FIELD = '' # Where to save extracted acronyms CFG_ACRON_FIELD = '' # ------------ bibclass config ------- # PARTIAL_TEXT # Marks the part of the fulltext to keep when running a partial match. # Each tuple contains the start and end percentages of a section. CFG_BIBCLASSIFY_PARTIAL_TEXT = ((0, 20), (40, 60)) # Format and output marcxml records in spires format CFG_SPIRES_FORMAT = False # The taxonomy used when no taxonomy is specified CFG_EXTRACTION_TAXONOMY = 'HEP' # WORD TRANSFORMATIONS # BibClassify creates a regular expression for each label found in the # ontology. # If the keyword belongs in 'INVARIABLE_WORDS', we return it whitout any # change. # If the keyword is found in 'EXCEPTIONS', we return its attached # regular expression. # If the keyword is matched by a regular expression of # 'UNCHANGE_REGULAR_EXPRESSIONS', we return the keyword without any # change. # At last, we perform the sub method of Python's re module using the # first element of the tuple as the regex and the second element as the # replacement string. # Regular expressions found here have been originally based on # Wikipedia's page on English plural. # [http://en.wikipedia.org/wiki/English_plural] CFG_BIBCLASSIFY_INVARIABLE_WORDS = ("any", "big", "chi", "der", "eta", "few", "low", "new", "non", "off", "one", "out", "phi", "psi", "rho", "tau", "two", "van", "von", "hard", "weak", "four", "anti", "zero", "sinh", "open", "high", "data", "dark", "free", "flux", "fine", "final", "heavy", "strange") CFG_BIBCLASSIFY_EXCEPTIONS = { "aluminium": r"alumini?um", "aluminum": r"alumini?um", "analysis": r"analy[sz]is", "analyzis": r"analy[sz]is", "behavior": r"behaviou?rs?", "behaviour": r"behaviou?rs?", "color": r"colou?rs?", "colour": r"colou?rs?", "deflexion": r"defle(x|ct)ions?", "flavor": r"flavou?rs?", "flavour": r"flavou?rs?", "gas": r"gas(s?es)?", "lens": r"lens(es)?", "matrix": r"matri(x(es)?|ces)", "muon": r"muons?", "neutrino": r"neutrinos?", "reflexion": r"refle(x|ct)ions?", "ring": r"rings?", "status": r"status(es)?", "string": r"strings?", "sum": r"sums?", "vertex": r"vert(ex(es)?|ices)", "vortex": r"vort(ex(es)?|ices)", } CFG_BIBCLASSIFY_UNCHANGE_REGULAR_EXPRESSIONS = ( re.compile("[^e]ed$"), re.compile("ics?$"), re.compile("[io]s$"), re.compile("ium$"), re.compile("less$"), re.compile("ous$"), ) # IDEAS # "al$" -> "al(ly)?" CFG_BIBCLASSIFY_GENERAL_REGULAR_EXPRESSIONS = ( (re.compile("ional"), r"ional(ly)?"), (re.compile("([ae])n(ce|t)$"), r"\1n(t|ces?)"), (re.compile("og(ue)?$"), r"og(ue)?s?"), (re.compile("([^aeiouyc])(re|er)$"), r"\1(er|re)s?"), (re.compile("([aeiouy])[sz]ation$"), r"\1[zs]ations?"), (re.compile("([aeiouy])[sz]ation$"), r"\1[zs]ations?"), (re.compile("([^aeiou])(y|ies)$"), r"\1(y|ies)"), (re.compile("o$"), r"o(e?s)?"), (re.compile("(x|sh|ch|ss)$"), r"\1(es)?"), (re.compile("f$"), r"(f|ves)"), (re.compile("ung$"), r"ung(en)?"), (re.compile("([^aiouy])s$"), r"\1s?"), (re.compile("([^o])us$"), r"\1(i|us(es)?)"), (re.compile("um$"), r"(a|ums?)"), ) # PUNCTUATION TRANSFORMATIONS # When building the regex pattern for each label of the ontology, ew also take # care of the non-alpha characters. Thereafter are two sets of transformations. # 'SEPARATORS' contains the transformation for the non-alpha characters that # can be found between two words. # 'SYMBOLS' contains punctuation that can be found at the end of a word. # In both cases, it the separator is not found in the dictionaries, we return # re.escape(separator) CFG_BIBCLASSIFY_SEPARATORS = { " ": r"[\s\n-]", "-": r"[\s\n-]?", "/": r"[/\s]?", "(": r"\s?\(", "*": r"[*\s]?", "- ": r"\s?\-\s", "+ ": r"\s?\+\s", } CFG_BIBCLASSIFY_SYMBOLS = { "'": r"\s?\'", } CFG_BIBCLASSIFY_WORD_WRAP = "[^\w-]%s[^\w-]" # MATCHING # When searching for composite keywords, we allow two keywords separated by one # of the component of 'VALID_SEPARATORS' to form a composite keyword. These # separators contain also the punctuation. CFG_BIBCLASSIFY_VALID_SEPARATORS = ( "of", "of a", "of an", "of the", "of this", "of one", "of two", "of three", "of new", "of other", "of many", "of both", "of these", "of each", "is", "the" ) # AUTHOR KEYWORDS # When looking for the keywords already defined in the document, we run the # following set of regex. CFG_BIBCLASSIFY_AUTHOR_KW_START = \ re.compile(r"(?i)key[ -]*words?[a-z ]*[.:] *") CFG_BIBCLASSIFY_AUTHOR_KW_END = ( re.compile(r"\n"), re.compile(r"\.\W"), re.compile(r"\sPACS"), re.compile(r"(?i)1[. ]*introduction\W"), re.compile(r"(?i)mathematics subject classification\W"), ) CFG_BIBCLASSIFY_AUTHOR_KW_SEPARATION = re.compile(" ?; ?| ?, ?| ?- ") # Modules to call to get output from them #CFG_EXTERNAL_MODULES = {'webtag' : 'call_from_outside'} CFG_EXTERNAL_MODULES = {} log = None _loggers = [] def get_logger(name): """Creates a logger for you - with the parent newseman logger and common configuration""" if log: logger = log.manager.getLogger(name) else: logger = logging.getLogger(name) hdlr = logging.StreamHandler(sys.stderr) formatter = logging.Formatter( '%(levelname)s %(name)s:%(lineno)d %(message)s') hdlr.setFormatter(formatter) logger.addHandler(hdlr) logger.setLevel(logging_level) logger.propagate = 0 if logger not in _loggers: _loggers.append(logger) return logger def set_global_level(level): global logging_level logging_level = int(level) for l in _loggers: l.setLevel(logging_level) log = get_logger('bibclassify') STANDALONE = False # Standalone mode has been removed. #try: # import invenio.legacy.search_engine #except: # STANDALONE = True # log.warning('Bibclassify is running in a standalone mode, access to database is not supported') if STANDALONE: import tempfile # try to find etcdir (first in this directory), and set etc to be one # level higher etcdir = ' ' bibetc = os.path.join(os.path.dirname(__file__), 'bibclassify') if os.path.isdir(bibetc) and os.access(bibetc, os.W_OK): etcdir = os.path.dirname(__file__) if not os.path.isdir(etcdir) or not os.access(etcdir, os.W_OK): etcdir = os.path.abspath( os.path.join(os.path.dirname(__file__), "../../../etc")) if not os.path.isdir(etcdir) or not os.access(etcdir, os.W_OK): etcdir = tempfile.gettempdir() log.warning("Setting CFG_CACHEDIR, CFG_WEBDIR, CFG_ETCDIR to: %s" % etcdir) # override a few special paths config.CFG_CACHEDIR = etcdir config.CFG_WEBDIR = etcdir config.CFG_ETCDIR = etcdir # shadow the config variables that bibclassify modules use -CFG_PREFIX = config.CFG_PREFIX CFG_CACHEDIR = config.CFG_CACHEDIR CFG_WEBDIR = config.CFG_WEBDIR CFG_ETCDIR = config.CFG_ETCDIR CFG_TMPDIR = config.CFG_TMPDIR # Redefine variable definitions if local config exists try: from invenio import bibclassify_config_local as localconf for confid in dir(localconf): if 'CFG' in confid: if hasattr(config, confid): log.info('Overriding global config %s with %s' % ( confid, getattr(localconf, confid))) setattr(config, confid, getattr(localconf, confid)) if confid in globals(): globals()[confid] = getattr(localconf, confid) log.info('Overriding bibclassify config %s with %s' % ( confid, getattr(localconf, confid))) except ImportError: # No local configuration was found. pass log.info('Initialized bibclassify config') diff --git a/invenio/legacy/bibformat/__init__.py b/invenio/legacy/bibformat/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/invenio/legacy/bibformat/adminlib.py b/invenio/legacy/bibformat/adminlib.py deleted file mode 100644 index ffe7073e3..000000000 --- a/invenio/legacy/bibformat/adminlib.py +++ /dev/null @@ -1,261 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""Handle requests from the web interface to configure BibFormat.""" - -__revision__ = "$Id$" - -import os -import re - -from invenio.modules.formatter.config import \ - CFG_BIBFORMAT_TEMPLATES_PATH, \ - CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION -from invenio.modules.formatter import engine as bibformat_engine - - -def get_outputs_that_use_template(filename): - """Return a list of output formats that call the given format template. - - The returned output formats also give their dependencies on tags. - We don't return the complete output formats but some reference to - them (filename + names):: - - [ {'filename':"filename_1.bfo" - 'names': {'en':"a name", 'fr': "un nom", 'generic':"a name"} - 'tags': ['710__a', '920__'] - }, - ... - ] - - :param filename: a format template filename - :return: output formats references sorted by (generic) name - """ - output_formats_list = {} - tags = [] - output_formats = bibformat_engine.get_output_formats(with_attributes=True) - for output_format in output_formats: - name = output_formats[output_format]['attrs']['names']['generic'] - # First look at default template, and add it if necessary - if output_formats[output_format]['default'] == filename: - output_formats_list[name] = {'filename':output_format, - 'names':output_formats[output_format]['attrs']['names'], - 'tags':[]} - # Second look at each rule - found = False - for rule in output_formats[output_format]['rules']: - if rule['template'] == filename: - found = True - tags.append(rule['field']) #Also build dependencies on tags - - # Finally add dependency on template from rule (overwrite default dependency, - # which is weaker in term of tag) - if found: - output_formats_list[name] = {'filename':output_format, - 'names':output_formats[output_format]['attrs']['names'], - 'tags':tags} - - - - keys = output_formats_list.keys() - keys.sort() - return map(output_formats_list.get, keys) - -def get_elements_used_by_template(filename): - """ - Returns a list of format elements that are called by the given format template. - The returned elements also give their dependencies on tags. - - Dependencies on tag might be approximative. See get_tags_used_by_element() - doc string. - - We must handle usage of bfe_field in a special way if we want to retrieve - used tag: used tag is given in "tag" parameter, not inside element code. - - The list is returned sorted by name:: - - [ {'filename':"filename_1.py" - 'name':"filename_1" - 'tags': ['710__a', '920__'] - }, - ... - ] - - :param filename: a format template filename - :return: elements sorted by name - """ - format_elements = {} - format_template = bibformat_engine.get_format_template(filename=filename, with_attributes=True) - code = format_template['code'] - format_elements_iter = bibformat_engine.pattern_tag.finditer(code) - for result in format_elements_iter: - function_name = result.group("function_name").lower() - if function_name is not None and function_name not in format_elements \ - and not function_name == "field": - filename = bibformat_engine.resolve_format_element_filename("BFE_"+function_name) - if filename is not None: - tags = get_tags_used_by_element(filename) - format_elements[function_name] = {'name':function_name.lower(), - 'filename':filename, - 'tags':tags} - elif function_name == "field": - # Handle bfe_field element in a special way - if function_name not in format_elements: - #Indicate usage of bfe_field if not already done - filename = bibformat_engine.resolve_format_element_filename("BFE_"+function_name) - format_elements[function_name] = {'name':function_name.lower(), - 'filename':filename, - 'tags':[]} - # Retrieve value of parameter "tag" - all_params = result.group('params') - function_params_iterator = bibformat_engine.pattern_function_params.finditer(all_params) - for param_match in function_params_iterator: - name = param_match.group('param') - if name == "tag": - value = param_match.group('value') - if not value in format_elements[function_name]['tags']: - format_elements[function_name]['tags'].append(value) - break - - keys = format_elements.keys() - keys.sort() - return map(format_elements.get, keys) - - -# Format Elements Dependencies -# - -def get_tags_used_by_element(filename): - """ - Returns a list of tags used by given format element - - APPROXIMATIVE RESULTS: the tag are retrieved in field(), fields() - and control_field() function. If they are used computed, or saved - in a variable somewhere else, they are not retrieved - @TODO: There is room for improvements. For example catch - call to BibRecord functions. - - :param filename: a format element filename - :return: tags sorted by value - """ - tags = {} - - format_element = bibformat_engine.get_format_element(filename) - if format_element is None: - return [] - elif format_element['type']=="field": - tags = format_element['attrs']['tags'] - return tags - - filename = bibformat_engine.resolve_format_element_filename(filename) - path = bibformat_engine.get_format_element_path(filename) - format = open(path, 'r') - code = format.read() - format.close - tags_pattern = re.compile(''' - (field|fields|control_field)\s* #Function call - \(\s* #Opening parenthesis - [\'"]+ #Single or double quote - (?P.+?) #Tag - [\'"]+\s* #Single or double quote - (,[^\)]+)* #Additional function param - \) #Closing parenthesis - ''', re.VERBOSE | re.MULTILINE) - - tags_iter = tags_pattern.finditer(code) - for result in tags_iter: - tags[result.group("tag")] = result.group("tag") - - return tags.values() - - -def get_templates_that_use_element(name): - """Return a list of format templates that call the given format element. - - The returned format templates also give their dependencies on tags:: - - [ {'filename':"filename_1.bft" - 'name': "a name" - 'tags': ['710__a', '920__'] - }, - ... - ] - - :param name: a format element name - :return: templates sorted by name - """ - format_templates = {} - tags = [] - files = os.listdir(CFG_BIBFORMAT_TEMPLATES_PATH) #Retrieve all templates - for possible_template in files: - if possible_template.endswith(CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION): - format_elements = get_elements_used_by_template(possible_template) #Look for elements used in template - format_elements = map(lambda x: x['name'].lower(), format_elements) - try: #Look for element - format_elements.index(name.lower()) #If not found, get out of "try" statement - - format_template = bibformat_engine.get_format_template(filename=possible_template, with_attributes=True) - template_name = format_template['attrs']['name'] - format_templates[template_name] = {'name':template_name, - 'filename':possible_template} - except: - pass - - keys = format_templates.keys() - keys.sort() - return map(format_templates.get, keys) - -# Output Formats Dependencies -# - -def get_templates_used_by_output(code): - """Return a list of templates used inside an output format give by its code. - - The returned format templates also give their dependencies on elements and tags:: - - [ {'filename':"filename_1.bft" - 'name': "a name" - 'elements': [{'filename':"filename_1.py", 'name':"filename_1", 'tags': ['710__a', '920__'] - }, ...] - }, - ... - ] - - :param code: outpout format code - :return: templates sorted by name - """ - format_templates = {} - output_format = bibformat_engine.get_output_format(code, with_attributes=True) - - filenames = map(lambda x: x['template'], output_format['rules']) - if output_format['default'] != "": - filenames.append(output_format['default']) - - for filename in filenames: - template = bibformat_engine.get_format_template(filename, with_attributes=True) - name = template['attrs']['name'] - elements = get_elements_used_by_template(filename) - format_templates[name] = {'name':name, - 'filename':filename, - 'elements':elements} - - - keys = format_templates.keys() - keys.sort() - return map(format_templates.get, keys) diff --git a/invenio/legacy/bibformat/bibreformat.py b/invenio/legacy/bibformat/bibreformat.py deleted file mode 100644 index 2d27e10e3..000000000 --- a/invenio/legacy/bibformat/bibreformat.py +++ /dev/null @@ -1,497 +0,0 @@ -# -*- mode: python; coding: utf-8; -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2010, 2011, 2012, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -from __future__ import print_function - -"""Call BibFormat engine and create HTML brief (and other) formats cache for - bibliographic records.""" - -__revision__ = "$Id$" - -import os -from intbitset import intbitset -from datetime import datetime - -from invenio.base.factory import with_app_context -from invenio.legacy.dbquery import run_sql -from invenio.legacy.search_engine import perform_request_search, search_pattern -from invenio.legacy.bibrank.citation_searcher import get_cited_by -from invenio.legacy.bibrank.citation_indexer import get_bibrankmethod_lastupdate -from invenio.legacy.bibformat.dblayer import save_preformatted_record -from invenio.utils.shell import split_cli_ids_arg -from invenio.modules.records.api import get_record -from invenio.legacy.bibsched.bibtask import task_init, \ - write_message, \ - task_set_option, \ - task_get_option, \ - task_update_progress, \ - task_has_option, \ - task_sleep_now_if_required -from invenio.modules.formatter.engine import format_record_1st_pass - - -def fetch_last_updated(fmt): - select_sql = "SELECT last_updated FROM format WHERE code = %s" - row = run_sql(select_sql, (fmt.lower(), )) - - # Fallback in case we receive None instead of a valid date - last_date = row[0][0] or datetime(year=1900, month=1, day=1) - - return last_date - - -def store_last_updated(fmt, iso_date): - sql = "UPDATE format SET last_updated = %s " \ - "WHERE code = %s AND (last_updated < %s or last_updated IS NULL)" - run_sql(sql, (iso_date, fmt.lower(), iso_date)) - - -# run the bibreformat task bibsched scheduled - -@with_app_context() -def bibreformat_task(fmt, recids, without_fmt, process): - """BibReformat main task. - - @param fmt: output format to use - @param process: - @param recids: a list of record IDs to reformat - @return: None - """ - write_message("Processing format %s" % fmt) - - t1 = os.times()[4] - - start_date = datetime.now().strftime("%Y-%m-%d %H:%M:%S") - - latest_bibrank_run = get_bibrankmethod_lastupdate('citation') - - def related_records(recids, recids_processed): - if fmt == "HDREF" and recids: - # HDREF represents the references tab - # the tab needs to be recomputed not only when the record changes - # but also when one of the citations changes - sql = """SELECT id, modification_date FROM bibrec - WHERE id in (%s)""" % ','.join(str(r) for r in recids) - - def check_date(mod_date): - return mod_date.strftime( - "%Y-%m-%d %H:%M:%S") < latest_bibrank_run - rel_recids = intbitset([recid for recid, mod_date in run_sql(sql) - if check_date(mod_date)]) - for r in rel_recids: - recids |= intbitset(get_cited_by(r)) - - # To not process recids twice - recids -= recids_processed - # Adds to the set of processed recids - recids_processed += recids - - return recids - - def recid_chunker(recids): - recids_processed = intbitset() - chunk = intbitset() - - for recid in recids: - if len(chunk) == 5000: - for r in related_records(chunk, recids_processed): - yield r - recids_processed += chunk - chunk = intbitset() - - if recid not in recids_processed: - chunk.add(recid) - - if chunk: - for r in related_records(chunk, recids_processed): - yield r - - recIDs = list(recid_chunker(recids)) - -### list of corresponding record IDs was retrieved -### now format the selected records - - if without_fmt: - write_message("Records to be processed: %d" % len(recIDs)) - write_message("Out of it records without existing cache: %d" % - len(without_fmt)) - else: - write_message("Records to be processed: %d" % len(recIDs)) - -### Initialize main loop - - total_rec = 0 # Total number of records - tbibformat = 0 # time taken up by external call - tbibupload = 0 # time taken up by external call - -### Iterate over all records prepared in lists I (option) - if process: - total_rec_1, tbibformat_1, tbibupload_1 = iterate_over_new(recIDs, fmt) - total_rec += total_rec_1 - tbibformat += tbibformat_1 - tbibupload += tbibupload_1 - -### Store last run time - if task_has_option("last"): - write_message("storing run date to %s" % start_date) - store_last_updated(fmt, start_date) - -### Final statistics - - t2 = os.times()[4] - - elapsed = t2 - t1 - message = "total records processed: %d" % total_rec - write_message(message) - - message = "total processing time: %2f sec" % elapsed - write_message(message) - - message = "Time spent on external call (os.system):" - write_message(message) - - message = " bibformat: %2f sec" % tbibformat - write_message(message) - - message = " bibupload: %2f sec" % tbibupload - write_message(message) - - -def check_validity_input_formats(input_formats): - """Check the validity of every input format. - - :param input_formats: list of given formats - :type input_formats: list - :return: if there is any invalid input format it returns this value - :rtype: string - """ - from invenio.modules.formatter import registry - tested_formats = set([aformat.lower() for aformat in input_formats]) - invalid_formats = tested_formats - set(registry.output_formats.keys()) - return invalid_formats[0] if len(invalid_formats) else '' - - -### Bibreformat all selected records (using new python bibformat) -### (see iterate_over_old further down) - - -def _update_recjson_format(recid, *args, **kwargs): - """Update RECJSON cache. - - :param int recid: record id to process - """ - dummy = get_record(recid, reset_cache=True) - - -def _update_format(recid, fmt): - """Usual format update procedure, gets the formatted record and saves it. - - :param int recid: record id to process - :param str fmt: format to update/create, i.e. 'HB' - """ - record, needs_2nd_pass = format_record_1st_pass(recID=recid, - of=fmt, - on_the_fly=True, - save_missing=False) - save_preformatted_record(recID=recid, - of=fmt, - res=record, - needs_2nd_pass=needs_2nd_pass, - low_priority=True) - - -_CFG_BIBFORMAT_UPDATE_FORMAT_FUNCTIONS = {'recjson': _update_recjson_format} -"""Specific functions to be used for each format if needed. -If not set `_update_format` will be used. -""" - - -def iterate_over_new(recIDs, fmt): - """Iterate over list of IDs. - - @param list: the list of record IDs to format - @param fmt: the output format to use - @return: tuple (total number of records, time taken to format, time taken - to insert) - """ - tbibformat = 0 # time taken up by external call - tbibupload = 0 # time taken up by external call - - tot = len(recIDs) - reformat_function = _CFG_BIBFORMAT_UPDATE_FORMAT_FUNCTIONS.get( - fmt.lower(), _update_format) - for count, recID in enumerate(recIDs): - t1 = os.times()[4] - reformat_function(recID, fmt) - t2 = os.times()[4] - tbibformat += t2 - t1 - if count % 100 == 0: - write_message(" ... formatted %s records out of %s" % - (count, tot)) - task_update_progress('Formatted %s out of %s' % (count, tot)) - task_sleep_now_if_required(can_stop_too=True) - - if tot % 100 != 0: - write_message(" ... formatted %s records out of %s" % (tot, tot)) - - return tot, tbibformat, tbibupload - - -def all_records(): - """Produce record IDs for all available records.""" - return intbitset(run_sql("SELECT id FROM bibrec")) - - -def outdated_caches(fmt, last_updated, chunk_size=5000): - sql = """SELECT br.id - FROM bibrec AS br - INNER JOIN bibfmt AS bf ON bf.id_bibrec = br.id - WHERE br.modification_date >= %s - AND bf.format = %s - AND bf.last_updated < br.modification_date - AND br.id BETWEEN %s AND %s""" - - last_updated_str = last_updated.strftime('%Y-%m-%d %H:%M:%S') - recids = intbitset() - max_id = run_sql("SELECT max(id) FROM bibrec")[0][0] or 0 - for start in xrange(1, max_id + 1, chunk_size): - end = start + chunk_size - recids += intbitset(run_sql(sql, (last_updated_str, fmt, start, end))) - - return recids - - -def missing_caches(fmt, chunk_size=100000): - """Produce record IDs to be formated, because their fmt cache is missing. - - @param fmt: format to query for - @return: record IDs generator without pre-created format cache - """ - write_message("Querying database for records without cache...") - - all_recids = intbitset() - max_id = run_sql("SELECT max(id) FROM bibrec")[0][0] or 0 - for start in xrange(1, max_id + 1, chunk_size): - end = start + chunk_size - sql = "SELECT id FROM bibrec WHERE id BETWEEN %s AND %s" - recids = intbitset(run_sql(sql, (start, end))) - sql = """SELECT id_bibrec FROM bibfmt - WHERE id_bibrec BETWEEN %s AND %s - AND format = %s""" - without_fmt = intbitset(run_sql(sql, (start, end, fmt))) - all_recids += recids - without_fmt - - return all_recids - - -def query_records(params): - """Produce record IDs from given query parameters. - - By passing the appriopriate CLI options, we can query here for additional - records. - """ - write_message("Querying database (records query)...") - res = intbitset() - if params['field'] or params['collection'] or params['pattern']: - - if not params['collection']: - # use search_pattern() whenever possible, as it can search - # even in private collections - res = search_pattern(p=params['pattern'], - f=params['field'], - m=params['matching']) - else: - # use perform_request_search when '-c' argument has been - # defined, as it is not supported by search_pattern() - res = intbitset(perform_request_search(req=None, - of='id', - c=params['collection'], - p=params['pattern'], - f=params['field'])) - return res - - -def task_run_core(): - """Run the task by fetching arguments from the BibSched task queue. - - This is what BibSched will be invoking via daemon call. - """ - fmts = task_get_option('format', 'HB,RECJSON') - for fmt in fmts.split(','): - last_updated = fetch_last_updated(fmt) - write_message("last stored run date is %s" % last_updated) - - recids = intbitset() - - if task_has_option("all"): - recids += all_records() - - if task_has_option("last"): - recids += outdated_caches(fmt, last_updated) - - if task_has_option('ignore_without'): - without_fmt = intbitset() - else: - without_fmt = missing_caches(fmt) - recids += without_fmt - - cli_recids = split_cli_ids_arg(task_get_option('recids', '')) - recids += cli_recids - - query_params = {'collection': task_get_option('collection', ''), - 'field': task_get_option('field', ''), - 'pattern': task_get_option('pattern', ''), - 'matching': task_get_option('matching', '')} - recids += query_records(query_params) - - bibreformat_task(fmt, - recids, - without_fmt, - not task_has_option('noprocess')) - - return True - - -def main(): - """Main that construct all the bibtask.""" - task_init(authorization_action='runbibformat', - authorization_msg="BibReformat Task Submission", - description=""" -BibReformat formats the records and saves the produced outputs for -later retrieval. - -BibReformat is usually run periodically via BibSched in order to (1) -format new records in the database and to (2) reformat records for -which the meta data has been modified. - -BibReformat has to be run manually when (3) format config files have -been modified, in order to see the changes in the web interface. - -Although it is not necessary to run BibReformat to display formatted -records in the web interface, BibReformat allows to improve serving -speed by precreating the outputs. It is suggested to run -BibReformat for 'HB' output. - -Option -m cannot be used at the same time as option -c. -Option -c prevents from finding records in private collections. - -Examples: - bibreformat Format all new or modified records (in HB and RECJSON). - bibreformat -o HD Format all new or modified records in HD. - bibreformat -o HD,HB Format all new or modified records in HD and HB. - - bibreformat -a Force reformatting all records (in HB). - bibreformat -c 'Photos' Force reformatting all records in 'Photos' collection (in HB). - bibreformat -c 'Photos' -o HD Force reformatting all records in 'Photos' collection in HD. - - bibreformat -i 15 Force reformatting record 15 (in HB). - bibreformat -i 15:20 Force reformatting records 15 to 20 (in HB). - bibreformat -i 15,16,17 Force reformatting records 15, 16 and 17 (in HB). - - bibreformat -n Show how many records are to be (re)formatted. - bibreformat -n -c 'Articles' Show how many records are to be (re)formatted in 'Articles' collection. - - bibreformat -oHB -s1h Format all new and modified records every hour, in HB. -""", help_specific_usage=""" -o, --formats \t Specify output format/s (default HB) - -n, --noprocess \t Count records to be formatted (no processing done) -Reformatting options: - -a, --all \t Force reformatting all records - -c, --collection \t Force reformatting records by collection - -f, --field \t Force reformatting records by field - -p, --pattern \t Force reformatting records by pattern - -i, --id \t Force reformatting records by record id(s) - --no-missing \t Ignore reformatting records without format -Pattern options: - -m, --matching \t Specify if pattern is exact (e), regular expression (r), - \t partial (p), any of the words (o) or all of the words (a) -""", - version=__revision__, - specific_params=("ac:f:p:lo:nm:i:", - ["all", - "collection=", - "matching=", - "field=", - "pattern=", - "format=", - "noprocess", - "id=", - "no-missing"]), - task_submit_check_options_fnc=task_submit_check_options, - task_submit_elaborate_specific_parameter_fnc= - task_submit_elaborate_specific_parameter, - task_run_fnc=task_run_core) - - -def task_submit_check_options(): - """Last checks and updating on the options...""" - if not (task_has_option('all') or task_has_option('collection') - or task_has_option('field') or task_has_option('pattern') - or task_has_option('matching') or task_has_option('recids')): - task_set_option('last', 1) - return True - - -def task_submit_elaborate_specific_parameter(key, value, opts, args): # pylint: disable-msg=W0613 - """ - Elaborate specific CLI parameters of BibReformat. - - @param key: a parameter key to check - @param value: a value associated to parameter X{Key} - @return: True for known X{Key} else False. - """ - if key in ("-a", "--all"): - task_set_option("all", 1) - elif key in ("--no-missing", ): - task_set_option("ignore_without", 1) - elif key in ("-c", "--collection"): - task_set_option("collection", value) - elif key in ("-n", "--noprocess"): - task_set_option("noprocess", 1) - elif key in ("-f", "--field"): - task_set_option("field", value) - elif key in ("-p", "--pattern"): - task_set_option("pattern", value) - elif key in ("-m", "--matching"): - task_set_option("matching", value) - elif key in ("-o", "--format"): - input_formats = value.split(',') - # check the validity of the given output formats - invalid_format = check_validity_input_formats(input_formats) - if invalid_format: - try: - raise Exception('Invalid output format.') - except Exception: # pylint: disable-msg=W0703 - from invenio.ext.logging import register_exception - register_exception( - prefix="The given output format '%s' is not available or " - "is invalid. Please try again" % - (invalid_format, ), alert_admin=True) - return - else: # every given format is available - task_set_option("format", value) - elif key in ("-i", "--id"): - task_set_option("recids", value) - else: - return False - return True - - -### okay, here we go: -if __name__ == '__main__': - main() diff --git a/invenio/legacy/bibformat/dblayer.py b/invenio/legacy/bibformat/dblayer.py deleted file mode 100644 index 66f2a7a53..000000000 --- a/invenio/legacy/bibformat/dblayer.py +++ /dev/null @@ -1,389 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""Database access related functions.""" - -import time -import zlib - -from invenio.legacy.dbquery import run_sql -from invenio.utils.date import localtime_to_utc - - -def get_creation_date(sysno, fmt="%Y-%m-%dT%H:%M:%SZ"): - """Return the creation date of the record 'sysno'. - - :param sysno: the record ID for which we want to retrieve creation date - :param fmt: output format for the returned date - :return: creation date of the record - @rtype: string - """ - out = "" - res = run_sql("SELECT DATE_FORMAT(creation_date, '%%Y-%%m-%%d %%H:%%i:%%s') FROM bibrec WHERE id=%s", (sysno,), 1) - if res[0][0]: - out = localtime_to_utc(res[0][0], fmt) - return out - -def get_modification_date(sysno, fmt="%Y-%m-%dT%H:%M:%SZ"): - """ - Returns the date of last modification for the record 'sysno'. - - :param sysno: the record ID for which we want to retrieve modification date - :param fmt: output format for the returned date - :return: modification date of the record - @rtype: string - """ - out = "" - res = run_sql("SELECT DATE_FORMAT(modification_date,'%%Y-%%m-%%d %%H:%%i:%%s') FROM bibrec WHERE id=%s", (sysno,), 1) - if res and res[0][0]: - out = localtime_to_utc(res[0][0], fmt) - return out - -# XML Marc related functions -def get_tag_from_name(name): - """ - Returns the marc code corresponding the given name - - :param name: name for which we want to retrieve the tag - :return: a tag corresponding to X{name} or None if not found - """ - res = run_sql("SELECT value FROM tag WHERE name LIKE %s", (name,)) - if len(res)>0: - return res[0][0] - else: - return None - -def get_tags_from_name(name): - """ - Returns the marc codes corresponding the given name, - ordered by value - - :param name: name for which we want to retrieve the tags - :return: list of tags corresponding to X{name} or None if not found - """ - res = run_sql("SELECT value FROM tag WHERE name LIKE %s ORDER BY value", (name,)) - if len(res)>0: - return list(res[0]) - else: - return None - -def tag_exists_for_name(name): - """ - Returns True if a tag exists for name in 'tag' table. - - :param name: name for which we want to check if a tag exist - :return: True if a tag exist for X{name} or False - """ - rows = run_sql("SELECT value FROM tag WHERE name LIKE %s", (name,)) - if len(rows) > 0: - return True - return False - -def get_name_from_tag(tag): - """ - Returns the name corresponding to a marc code - - :param tag: tag to consider - :return: a name corresponding to X{tag} - """ - res = run_sql("SELECT name FROM tag WHERE value LIKE %s", (tag,)) - if len(res)>0: - return res[0][0] - else: - return None - -def name_exists_for_tag(tag): - """ - Returns True if a name exists for tag in 'tag' table. - - :param tag: tag for which we want to check if a name exist - :return: True if a name exist for X{tag} or False - """ - rows = run_sql("SELECT name FROM tag WHERE value LIKE %s", (tag,)) - if len(rows) > 0: - return True - return False - -def get_all_name_tag_mappings(): - """ - Return the list of mappings name<->tag from 'tag' table. - - The returned object is a dict with name as key (if 2 names are the same - we will take the value of one of them, as we cannot make the difference in format - templates) - - :return: a dict containing list of mapping in 'tag' table - """ - out = {} - query = "SELECT value, name FROM tag" - res = run_sql(query) - for row in res: - out[row[1]] = row[0] - return out - - -# Output formats related functions - -def get_output_format_id(code): - """ - Returns the id of output format given by code in the database. - - Output formats are located inside 'format' table - - :param code: the code of an output format - :return: the id in the database of the output format. None if not found - """ - f_code = code - if len(code)>6: - f_code = code[:6] - res = run_sql("SELECT id FROM format WHERE code=%s", (f_code.lower(),)) - if len(res)>0: - return res[0][0] - else: - return None - -def add_output_format(code, name="", description="", content_type="text/html", visibility=1): - """ - Add output format into format table. - - If format with given code already exists, do nothing - - :param code: the code of the new format - :param name: a new for the new format - :param description: a description for the new format - :param content_type: the content_type (if applicable) of the new output format - :param visibility: if the output format is shown to users (1) or not (0) - :return: None - """ - output_format_id = get_output_format_id(code) - if output_format_id is None: - query = "INSERT INTO format SET code=%s, description=%s, content_type=%s, visibility=%s" - params = (code.lower(), description, content_type, visibility) - run_sql(query, params) - set_output_format_name(code, name) - -def remove_output_format(code): - """ - Removes the output format with 'code' - - If code does not exist in database, do nothing. - The function also removes all localized names in formatname table. - - :param code: the code of the output format to remove - :return: None - """ - output_format_id = get_output_format_id(code) - if output_format_id is None: - return - - query = "DELETE FROM formatname WHERE id_format='%s'" % output_format_id - run_sql(query) - query = "DELETE FROM format WHERE id='%s'" % output_format_id - run_sql(query) - -def get_output_format_description(code): - """Return the description of the output format given by code. - - If code or description does not exist, return empty string. - - :param code: the code of the output format to get the description from - :return: output format description - """ - - res = run_sql("SELECT description FROM format WHERE code=%s", (code,)) - if len(res) > 0: - res = res[0][0] - if res is not None: - return res - return "" - -def set_output_format_description(code, description): - """ - Sets the description of an output format, given by its code - - If 'code' does not exist, create format - - :param code: the code of the output format to update - :param description: the new description - :return: None - """ - output_format_id = get_output_format_id(code) - if output_format_id is None: - add_output_format(code, "", description) - - query = "UPDATE format SET description=%s WHERE code=%s" - params = (description, code.lower()) - run_sql(query, params) - -def get_output_format_visibility(code): - """ - Returns the visibility of the output format, given by its code - - If code does not exist, return 0. - - :param code: the code of an output format - :return: output format visibility (0 if not visible, 1 if visible - """ - res = run_sql("SELECT visibility FROM format WHERE code=%s", (code,)) - if len(res) > 0: - res = res[0][0] - if res is not None and int(res) in range(0, 2): - return int(res) - return 0 - -def set_output_format_visibility(code, visibility): - """ - Sets the visibility of an output format, given by its code - - If 'code' does not exist, create format - - :param code: the code of the output format to update - :param visibility: the new visibility (0: not visible, 1:visible) - :return: None - """ - output_format_id = get_output_format_id(code) - if output_format_id is None: - add_output_format(code, "", "", "", visibility) - - query = "UPDATE format SET visibility=%s WHERE code=%s" - params = (visibility, code.lower()) - run_sql(query, params) - - -def set_output_format_content_type(code, content_type): - """ - Sets the content_type of an output format, given by its code - - If 'code' does not exist, create format - - :param code: the code of the output format to update - :param content_type: the content type for the format - :return: None - """ - output_format_id = get_output_format_id(code) - if output_format_id is None: - # add one if not exist (should not happen) - add_output_format(code, "", "", content_type) - - query = "UPDATE format SET content_type=%s WHERE code=%s" - params = (content_type, code.lower()) - run_sql(query, params) - - -def set_output_format_name(code, name, lang="generic", type='ln'): - """Set the name of an output format given by code. - - If 'type' different from 'ln' or 'sn', do nothing. - If 'name' exceeds 256 chars, 'name' is truncated to first 256 chars. - If 'code' does not correspond to exisiting output format, create format. - If "generic" is given has lang. - - The localized names of output formats are located in formatname table. - - :param code: the code of an ouput format - :param type: either 'ln' (for long name) and 'sn' (for short name) - :param lang: the language in which the name is given - :param name: the name to give to the output format - :return: None - """ - - if len(name) > 256: - name = name[:256] - if type.lower() != "sn" and type.lower() != "ln": - return - output_format_id = get_output_format_id(code) - if output_format_id is None and lang == "generic" and type.lower() == "ln": - # Create output format inside table if it did not exist - # Happens when the output format was added not through web interface - add_output_format(code, name) - output_format_id = get_output_format_id(code) # Reload id, because it was not found previously - - if lang =="generic" and type.lower()=="ln": - # Save inside format table for main name - query = "UPDATE format SET name=%s WHERE code=%s" - params = (name, code.lower()) - run_sql(query, params) - else: - # Save inside formatname table for name variations - run_sql("REPLACE INTO formatname SET id_format=%s, ln=%s, type=%s, value=%s", - (output_format_id, lang, type.lower(), name)) - -def change_output_format_code(old_code, new_code): - """Change the code of an output format. - - :param old_code: the code of the output format to change - :param new_code: the new code - :return: None - """ - output_format_id = get_output_format_id(old_code) - if output_format_id is None: - return - - query = "UPDATE format SET code=%s WHERE id=%s" - params = (new_code.lower(), output_format_id) - run_sql(query, params) - -def get_preformatted_record(recID, of, decompress=zlib.decompress): - """Return the preformatted record with id 'recID' and format 'of'. - - Note that second item in tuple is True if we need a 2nd pass. - If corresponding record does not exist for given output format, - returns None. - - :param recID: the id of the record to fetch - :param of: the output format code - :param decompress: the method used to decompress the preformatted record in database - :return: formatted record as String, or None if not exist - """ - # Decide whether to use DB slave: - if of in ('xm', 'recstruct'): - run_on_slave = False # for master formats, use DB master - else: - run_on_slave = True # for other formats, we can use DB slave - # Try to fetch preformatted record - query = """SELECT value, needs_2nd_pass FROM bibfmt - WHERE id_bibrec = %s AND format = %s""" - params = (recID, of) - res = run_sql(query, params, run_on_slave=run_on_slave) - if res: - value = decompress(res[0][0]) - needs_2nd_pass = bool(res[0][1]) - # record 'recID' is formatted in 'of', so return it - return value, needs_2nd_pass - else: - return None, None - - -def save_preformatted_record(recID, of, res, needs_2nd_pass=False, - low_priority=False, compress=zlib.compress): - """Store preformated record in the database.""" - start_date = time.strftime('%Y-%m-%d %H:%M:%S') - formatted_record = compress(res) - sql_str = "" - if low_priority: - sql_str = " LOW_PRIORITY" - run_sql("""INSERT%s INTO bibfmt - (id_bibrec, format, last_updated, value, needs_2nd_pass) - VALUES (%%s, %%s, %%s, %%s, %%s) - ON DUPLICATE KEY UPDATE - last_updated = VALUES(last_updated), - value = VALUES(value), - needs_2nd_pass = VALUES(needs_2nd_pass) - """ % sql_str, - (recID, of, start_date, formatted_record, needs_2nd_pass)) diff --git a/invenio/legacy/bibformat/doc/admin/bibformat-admin-guide.webdoc b/invenio/legacy/bibformat/doc/admin/bibformat-admin-guide.webdoc deleted file mode 100644 index 5dacf140e..000000000 --- a/invenio/legacy/bibformat/doc/admin/bibformat-admin-guide.webdoc +++ /dev/null @@ -1,1189 +0,0 @@ -# -*- mode: html; coding: utf-8; -*- - -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - - - - - -

Contents

- - - - - - -

1. Overview

-

1.1 How BibFormat Works

-

BibFormat is in charge of formatting the bibliographic records that -are displayed to your users. It is called by the search engine when it has to -format a record.

- -

As you might need different kind of formatting depending -on the type of record, but potentially have a huge amount of records in your database, you cannot specify -for each of them how they should look. Instead BibFormat uses a rule-based decision process -to decide how to format a record.
-The best way to understand how BibFormat works is to have a look at -a typical workflow:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Step 1:
- Use output format HD - When Invenio has to display a record, it - asks BibFormat to format the record with the given output format - and language. For example here the requested output format is - hd, which is a short code - for "HTML Detailed". This means that somehow a user arrived on - the page of the record and asked for a detailed view of the - record.
Step 2:
1. Use Template [Picture HTML Detailed] if tag [980__a] is equal to [PICTURE] 2. Use Template [Thesis HTML detailed] if tag [980__a] is equal to [THESIS] 3. By default use [Default HTML Detailed] - - - Beside is a screenshot of the "hd" or "HTML Detailed" output format. - You can see that the output format does not specify how to format the record, but - contains a set of rules which define which template must be used.
- The rules are evaluated from top to bottom. - Each rule defines a condition on a field of the record, and a format template to use to - format the record if the condition matches. - Let's say that the field 980__a of the record is equal to - "Picture". Then first rules matches, and format template - Picture HTML Detailed is - used for formatting by BibFormat.
- You can add, remove or edit output formats here
Step 3:
-
- <h1 align="center"><BFE_MAIN_TITLE/></h1>
- <p align="center">
- <BFE_AUTHORS separator="; " - link="yes"/><br/>
- <BFE_DATE format="%d %B %Y"> .- <BFE_NB_PAGES suffix="p">
- </p>
We see an extract of the Picture HTML Detailed format on the right, - as it is shown in the template editor. As you can see it - is mainly written using HTML. There are however some tags that - are not part of standard HTML. Those tags that starts with - <BFE_ are placeholders for the record values. For - example <BFE_MAIN_TITLE/> tells BibFormat to write the title - of the record. We call these tags "elements". Some - elements have parameters. This is the case of the <BFE_AUTHORS> element, - which can take separator and link as - parameters. The value of separator will be used to separate - authors' names and the link parameter tells if links to authors' - websites have to be created. - All elements are described in the elements documentation.
- You can add, remove or edit format templates here. -

- In addition to this modified HTML language, BibFormat also supports XSL stylesheets as format templates. - Read the XSL Format Templates section to learn more about XSLT support for your format templates. -

-
Step 4:
- def format_element(bfo, separator='; ', link='no'):
-    """
-    Prints the list of authors for the record

-    @param separator a character to separate the authors
-    @param link if 'yes' print HTML links to authors
-    """
-    authors = bfo.fields("100__a")
-    if link == 'yes':
-       authors = map(lambda x: '<a href="'+CFG_SITE_URL+'/search?f=author&p='\
-                   - + quote(x) +'">'+x+'</a>', authors)
-    return authors.split(separator) -
A format element is written in Python. It acts as a bridge - between the record in the database and the format - template. Typically you will not have to write or read format - elements, just call them from the templates. Each element outputs - some text that is written in the template where it is called.
- Developers can add new elements by creating a new file, naming it - with the name of element, and write a Python format_element - function that takes as parameters the parameters of the elements - plus a special one bfo. Regular Python code can be - used, including import of other modules.
-
- -

In summary BibFormat is called by specifying a record and an output -format, which relies on different templates to do the formatting, and -which themselves rely on different format elements. Only developers need to modify -the format elements layer.

- - - - - - - - - - - - - - - - -
Output Format
Template
Template
Format Element
Format Element
Format Element
Format Element
-

You should now understand the philosophy behind BibFormat.

- -

1.2 Short Tutorial

-

Let's try to create our own format. -This format will just print the title of a record. -

- -

First go to the main BibFormat admin page. -Then click on the "Manage Ouput Format" links. You will see the list of all output formats:

-Output formats management page -

This is were you can delete, create or check output formats. -The menu at the top of the page let you go to other admininistration pages.
-Click on the "Add New Output Format" button at the bottom of the page. You can then fill in some attributes -for the output format. Choose "title" as code, "Only Title" as name and "Prints only title" as description:

-Screenshot of the Update Output Format Attributes page -

Leave other fields blank, and click on the button "Update Output format Attributes".
You are then -redirected to the rules editor. Notice the menu at the top which let you close the editor, change the attributes again -and check the output format. However do not click on these links before saving your modification of rules!

-Output format menu -

As our format does not need to have a different behaviour depending on the record, we do not need to add new rules to the format. You just need to select a format template in the "By default use" list. However we first have to create our special format template that only print titles. So close the editor using the menu at the top of the page, and in the menu that just appeared instead, click on "Manage Format Templates". In a similar way to output formats, you see the list of format templates.

-Format template management page -

-Click on the "Add New Format Template" button at the bottom of the page. As for the output format, fill in the attributes of the template with name "Title" and any relevant description. -

- -update format template attributes -

Click on the "Update Output Format Attributes" button. You are redirected to the template editor. The editor is divided in three parts. The upper left part contains the code of the template. The bottom part is a preview of the template. The part on the right side is a short remainder of the format elements you can use in you template. You can hide this documentation by clicking on "Hide Documentation".

-Format template editor -

The above screenshot shows the template code already filled in. It calls the BFE_TITLE element. If you do not know the name of the element you want to call, you can search for it using the embedded documentation search. You can try to add other elements into your template, or write some HTML formatting.

-

When you are satisfied with your template, click on the save button, close the editor and go back to the "Only titles" output format rules editor. There select the template you have just created in the "Use by default" menu and save the ouput format and you are done.

-

This tutorial does not cover all aspects of the management of formats. It also does not show all the power of output formats, as the one we have created simply call a template. However you have seen enough to configure BibFormat trough the web interface. Read the sections below to learn more about it.

- -

1.3 Administer Through the Web Interface or Through the Configuration files

-

BibFormat can be administered in two ways. The first way is to use the provided web interface. It should be the most -convenient way of doing for most users. The web interface is simple to use and provides great tools to manage your formats. Its only limitation concerns the format elements, which cannot be modified using it (But the web interface provide a dynamically generated documentation of your elements).
-The other way to administer BibFormat is to directly modify the configuration files using your preferred text editor. This way of doing can bring much power to advanced users, but requires an access to the server's files. It also requires that the user double-check his modifications, or use the web interface to ensure the validity and correctness of his formats.

-

In this manual we will show both ways of doing. For each explication we show first how to do it through the web interface, then how to do it by manipulating the configuration files. Non-power users can stop reading as soon as they encounter the text "For developers and adventurers only".

-

We generally recommend to use the web interface, excepted for writing -format elements.

- - -

2. Output Formats

-

As you potentially have a huge amount of -bibliographic records, you cannot specify manually for each of them -how it should be formatted. This is why you can define rules that will -allow BibFormat to understand which kind of formatting to apply to a given -record. You define this set of rules in what is called an "output -format".

- -

You can have different output formats, each with its own characteristics. -For example you certainly want that when multiple bibliographic records are -displayed at the same time (as it happens in search results), only -short versions are shown to the user , while a detailed record is -preferable when a single record is displayed, whatever the type of the record.
-You might also want to -let your users decide which kind of output they want. For example you -might need to display HTML for regular web browsing, but would also -give a BibTeX version of the bibliographic reference for direct -inclusion in a LaTeX document.

-

To summarize, an output format groups similar kind of formats, specifying which kind -of formatting has to be done, but not how it has to be done.

- -

2.1 Add an Output Format

-

To add a new output format, go to the Manage Output Formats page and click on the "Add New Output Format" button at the bottom of the page. The format has been created. You can then specify the attributes of the output format. See Edit the Attributes of an Output Format to learn more about it.

- For developers and adventurers only: -

Alternatively you can directly add a new output format file into the - /etc/bibformat/outputs/ directory of your Invenio installation, if you have - access to the server's files. Use the format extension .bfo for your file.

-

You should also check that user www-data has read/write access to the file, - if you want to be able to modify the rules through the web interface.

- -

2.2 Remove an Output Format

-

To remove an output format, go to the Manage Output Formats page and click on the "Delete" button facing the output format you want to delete. If you cannot click on the button (the button is not enabled), this means that you do not have sufficent priviledge to do so (Format is protected. Contact the administrator of the system).

-For developers and adventurers only: -

You can directly remove an output format from the /etc/bibformat/outputs/ directory of your Invenio installation. -However you must make sure that it is removed from the tables format and formatname in the database, so that other modules know that it is not longer available.

- -

2.3 Edit the Rules of an Output Format

-

When you create a new output format, you can at first only specify the default template, - that is the one which is used when all rules fail. In the case of a basic output format, - this is enough. You can however add other rules, by clicking on the "Add New Rule" button.
- Once you have added a rule, you can fill it with a condition, and a template that should be used - if the condition is true. For example the rule

- Rule: Use template [Picture HTML Detailed] if field [980__a] is equal to [PICTURE] -

will use template named "Picture HTML Detailed" if the field 980__a of the record to format is equal to "Picture". - Note that text "PICTURE" will match any letter case like "picture" or "Picture". - Leading and trailing spaces are ignored too (" Picture " will match "PICTURE"). -
Tips: you can use a regular expression as text. For example "PICT.*" will match "pictures" - and "PICTURE".

- -

Reorder rules using arrows
- The above configuration will use format template "Default HTML Detailed" if all above rules fail (in that case - if field 980__a is different from "PICTURE"). If you have more rules, you decide in which order the conditions are evaluated. You can reorder rules by clicking on the small arrows on the left of the rules. -

-

Note that when you are migrating your output formats from the old PHP BibFormat, you might not have translated all the formats to which your output formats refers. In that case you should use defined in old BibFormat option in the format templates menu, to make BibFormat understand that a match for this rule must trigger a call to the Behaviour of the old BibFormat. See section on Run old and new formats side by side for more details on this.

- For developers and adventurers only: -

To write an output format, use the following syntax:
- First you - define which field code you put as the conditon for the rule. - You suffix it with a column. Then on next lines, define the values of - the condition, followed by --- and then the filename of the template - to use:

-
-  tag 980__a:
-  PICTURE --- PICTURE_HTML_BRIEF.bft
-  PREPRINT --- PREPRINT_HTML_BRIEF.bft
-  PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
-
-

- This means that if value of field 980__a is equal to PICTURE, then we - will use format template PICTURE_HTML_BRIEF.bft. Note that you must - use the filename of the template, not the name. Also note that spaces - at the end or beginning are not considered. On the following lines, - you can either put other conditions on tag 980__a, or add another tag on - which you want to put conditions.

-

At the end you can add a default condition:

-
-   default: PREPRINT_HTML_BRIEF.bft
-
-

which means that if no condition is matched, a format suitable for - Preprints will be used to format the current record.

- -

The output format file could then look like this:

-
-  tag 980__a:
-  PICTURE --- PICTURE_HTML_BRIEF.bft
-  PREPRINT --- PREPRINT_HTML_BRIEF.bft
-  PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
-
-  tag 8560_f:
-  .*@cern.ch --- SPECIAL_MEMBER_FORMATTING.bft
-
-  default: PREPRINT_HTML_BRIEF.bft
-
-

You can add as many rules as you want. Keep in mind that they are read - in the order they are defined, and that only first rule that - matches will be used. - Notice the condition on tag 8560_f: it uses a regular expression to - match any email address that ends with @cern.ch (the regular - expression must be understandable by Python) -

-

2.4 Edit the Attributes of an Output Format

-

An output format has the following attributes: -

    -
  • code: a short identifier that is used to identify the output format. It must be unique and contain a maximum of 6 letters. Note that the code is not case sensitive ("HB" is equal to "hb").
    • -
  • content type: this is the content type of the format, specified in Mime. For example if you were to produce an Excel output, you could use application/ms-excel as content type. If a content type is specified, Invenio will not print the usual header and footerfor the page, but will trigger a download in the client's browser when viewing the page (Unless the browser handles this content type).
    • -
  • name: a generic name to display in the interface for this output format.
    • -
  • (*) name: internationalized names for the output format, used for displaying localized name in the search interface.
    • -
  • description: an optional description for the output format.
    • -

-

Please read this information regarding output format codes: - There are some reserved codes that you should not use, or at least be aware of when choosing a code for your - output format. The table below summarizes these special words: -

-

- - - - - - - - - - - - - - - -
CodePurpose
HBUsed for displaying list of results of a search.
HDUsed when no format is specified when viewing a record.
HMUsed for Marc output. The format is special in the sense that it filters - fields to display according to the 'ot' GET parameter of the HTTP request.
Starting with letter 't'Used for displaying the value of the field specified by the 'ot' GET parameter of the HTTP request.
Starting with 3 digitsUsed for displaying the value of the field specified by the digits.
- -

- For developers and adventurers only: -

Excepted for the code, output format attributes cannot be changed in the output format file. These - attributes are saved in the database. As for the code, it is the name of the output format file, - without its .bfo extension. If you change this name, do not forget to propagate the modification in the database.

-

2.5 Check the Dependencies an Output Format

-

To check the dependencies of an output format on format templates, format elements and tags, - go to the Manage Output Formats page, click on - the output format you want to check, and then in the menu click on "Check Dependencies".

- Check Dependencies menu -

The next page shows you: -

  • the format templates which might be called by the rules of the output format
    • -
  • the elements used in each of these templates
    • -
  • the Marc tags involved in these elements
    • -
- Note that some Marc tags might be omitted.

-

2.6 Check the Validity an Output Format

-

To check the validity of an output format, simply go to the Manage Output Formats page, and look at the column 'status' for the output format you want to check. If message "Ok" is there, - then no problem was found with the output format. If message 'Not Ok' is in the column, click on it to see - the problems that have been found for the output format.

- -

3. Format Templates

-

A format template defines how a record should be formatted. For example it specifies which fields of the record are to be displayed, in which order and with which visual attributes. Basically the format template is written in HTML, so that it is easy for anyone to edit it. BibFormat also has support for XSLT for formatting. Read more about XSL format templates here.

-

3.1 Add a Format Template

-

To add a new format template, go to the Manage Format Templates page and click on the "Add New Format Template" button at the bottom of the page. The format has been created. You can then specify the attributes of the format template, or ask to make a copy of an existing format. - See Edit the Attributes of a Format Template to learn more about editing the attributes.

- For developers and adventurers only: -

Alternatively you can directly add a new format template file into the - /etc/bibformat/format_templates/ directory of your Invenio installation, if you have - access to the server's files. Use the format extension .bft for your file.

-

You should also check that user www-data has read/write access to the file, - if you want to be able to modify the code and the attributes of the template through the web interface.

- -

3.2 Remove a Format Template

-

To remove a format template, go to the Manage Format Templates page and click on the "Delete" button facing the format template you want to delete. If you cannot click on the button (the button is not enabled), this means that you do not have sufficent priviledge to do so (Format is protected. Contact the administrator of the system).

-For developers and adventurers only: -

You can directly remove the format template from the /etc/bibformat/format_templates/ directory of your Invenio installation.

-

3.3 Edit the Code of a Format Template

-

You can change the formatting of records by modifying the code of a template. -

To edit the code of a format template - go to the Manage Format Templates page. Click on - the format template you want to edit to load the template editor.

- -

The format template editor contains three panels. The left upper panel is the code editor. This is were - you write the code that specifies the formatting of a template. The right-most panel is a short documentation - on the "bricks" you can use in your format template code. The panel at the bottom of the page allows you to preview the template.

- Template Editor Page -

The following sections explain how to write the code that specifies the formatting.

-

3.4 Basic Editing

-

The first thing you have to know before editing the code is that everything you write in the - code editor is printed as such by BibFormat. Well almost everything (as you will discover later).

-

For example if you write "My Text", then for every record the output will be "My Text". Now let's say - you write "<b>My Text</b>": the output will still be "<b>My Text</b>", but as we display in a web browser, it will look like - "My Text" (The browser interprets the text inside tags <b></b> as "bold". Also note that the look may depend on the CSS style of your page).

-

Basically it means that you can write HTML to do the formatting. If you are not experienced with HTML you can use an HTML editor to create your layout, and the copy-paste the HTML code inside the template.

-

Do not forget to save your work by clicking on the save button before you leave the editor!

- For developers and adventurers only: -

- You can edit the code of a template using exactly the same syntax as in the web interface. The code of the template - is in the template file located in the /etc/bibformat/format_templates/ directory of your Invenio installation. You just - have to take care of the attributes of the template, which are saved in the same file as the code. See Edit the Attributes of a Format Template to learn more about it. -

- -

3.5 Use Format Elements

-

To add a dynamic behaviour to your format templates, that is display for example a different title - for each record or a different background color depending on the type of record, you can use the format elements.

-

Format elements are the smart bricks you can copy-paste in your code to get the attributes of template - that change depending on the record. A format element looks like a regular HTML tag.

-

For example, to print - the title of a record, you can write <BFE_TITLE /> in your template code where you want to diplay the title

-

Format elements can take values as parameters. This allows to customize the behaviour of an element. For example you can write <BFE_TITLE prefix="Title: " />, and BibFormat will take care of printing the title for you, with prefix "Title: ". The difference between Title: <BFE_TITLE /> and <BFE_TITLE prefix="Title: " /> is that the first option will always write "Title: " while the second one will only print "Title: " if there exist a title for the record in the database. Of course there are chances that there is always a title for each record, but this can be useful for less common fields.

-

Some parameters are available for all elements. This is the case for the following ones: -

    -
  • prefix: a prefix printed only if the record has a value for the element.
    • -
  • suffix: a suffix printed only if the record has a value for the element.
    • -
  • default: a default value printed if the record has no value for the element. In that case prefix and suffix are not printed.
    • -
-

-

Some parameters are specific to elements. To get information on all available format elements you can read the Format Elements Documentation, which is generated dynamically for all existing elements. it will show you what the element do and what parameters it can take.

-

While format elements looks like HTML tags, they differ in the followings ways from traditional ones: -

    -
  • A format element is a single tag: you cannot have <BFE_TITLE >some text<BFE_TITLE /> but only <BFE_TITLE />.
    • -
  • The values of the parameters accept any characters, including < and >. The only limitation is that you cannot use the type of quotes that delimit that value: you can have for example <BFE_TITLE someParam="a lot of single quotes ' ' ' ' "/> or <BFE_TITLE someParam='a lot of double quotes " " " '/>, but not <BFE_TITLE someParam="a lot of same quotes as delimiter " " " "/>.
    • -
  • Format elements names always start with BFE_.
    • -
  • Format element can expand on multiple lines.
    • -
-

-

Tips: you can use the special element <BFE_FIELD tag="" /> to print the value - of any field of a record in your templates. This practice is however not - recommended because it would necessitate to revise all format - templates if you did change the meaning of the MARC code schema.

- -

3.6 Preview a Format Template

-

To preview a format template go to the Manage Format Templates page and click on the format template you want to preview to open the template editor. The editor contains a preview panel at the bottom of the page.

- Preview Panel -

Simply click on " Reload Preview" button to preview the template (you do not need to save the code before previewing).
- Use the "Language" menu to preview the template in a given language

-

You can fill in the "Search Pattern" field to preview a specific record. The search pattern uses exactly the same - syntax as the one used in the web interface. The only difference with the regular search engine is that only the first matching record is shown.

- For developers and adventurers only: -

If you do not want to use the web interface to edit the templates but still would like to get previews, you can open the preview frame of any format in a new window/tab. In this mode you get a preview of the template (if it is placed in the /etc/bibformat/format_templates/ directory of your Invenio installation). The parameters of the preview are specified in the url:

    -
  • bft: the filename of the format template to preview
    • -
  • ln: the language to use for the preview
    • -
  • pattern_for_preview: the search pattern to use for the preview
    • -

-

3.7 Internationalization (i18n)

-

You can add translations to your format templates. To do so enclose the text you want to localize - with tags corresponding to the two letters of the language. For example if we want to localize "title", write <en>Title</en>. Repeat this for each language in which you want to make "title" available: <en>Title</en><fr>Titre</fr><de>Titel</de>. - Finally enclose everything with <lang> </lang> tags: <lang><en>Title</en><fr>Titre</fr><de>Titel</de></lang>

-

For each <lang> group only the text in the user's language is displayed. If user's language is not -available in the <lang> group, your default Invenio language is used.

- -

3.8 Escaping special HTML/XML characters

- -

By default, BibFormat escapes all values returned by format -elements. As a format template designer, you can assume in almost all -cases that the values you get from a format element will be escaped -for you. For special cases, you can set the parameter -escape of the element to '0' when calling it, to make -BibFormat understand that it must not escape the values of the -element, or to '1' to force the escaping.

-See the complete list of escaping modes. -

-For example -<bfe_abstract /> will return:
- -[...]We find that for spatially-flat cosmologies, background lensing
-clusters with reasonable mass-to-light ratios lying in the
-redshift range 0&lt;1 are strongly excluded, [...]

-while <bfe_abstract escape="0"/> will return:
-[...]We find that for spatially-flat cosmologies, background lensing
-clusters with reasonable mass-to-light ratios lying in the
-redshift range 0<1 are strongly excluded, [...]

-

-

In most cases, you will not set escape to 1, nor 0, but -just let the developer of the element take care of that for you.

- -

Please note that values given in special parameters -prefix, suffix, default and -nbMax are never escaped, whatever the value of -escape is (but other parameters will). You have to take -care of that in your format template, as well as of all other values that -are not returned by the format elements.

- -

3.9 Edit the Attributes of a Format Template

-

To edit the attributes of a format template - go to the Manage Format Templates page, click on - the format template you want to edit, and then in the menu click on "Modify Template Attributes".

-

- A format template contains two attributes: -

    -
  • Name: the name of the template
    • -
  • Description: a short description of the template
    • -
-

Note that changing these parameters has no impact on the formatting. Their purpose in only to - document the template.

-

If the name you have chosen already exists for another template, you name will be suffixed with an integer so that the name is unique.

-

You should also be aware that if you change the name of a format template, all output formats that were linking to this template will be changed to match the new name.

- For developers and adventurers only: -

You can change the attributes of a template by editing its file in the /etc/bibformat/format_templates/ directory of your Invenio installation. The attributes must be enclosed with tags <name> </name> and <description> </description> and should ideally be placed at the beginning of the file.

-

Also note that the admin web interface tries to keep the name of the template in sync with the filename of the template. If the name is changed through the web interface, the filename of the template is changed, and all output formats that use this template are updated. You have to do update output formats manually if you change the filename of the template without the web interface.

- -

3.10 Check the Dependencies of a Format Template

-

To check the dependencies of a format template - go to the Manage Format Template page, click on - the format template you want to check, and then in the menu click on "Check Dependencies".

- Check Dependencies menu -

The next page shows you: -

  • The output formats that use this format template
    • -
  • the elements used in the template (and Marc tags use in these elements in parentheses)
    • -
  • A summary of all the Marc tags involved in the elements of the template
    • -
- Note that some Marc tags might be omitted.

-

3.11 Check the Validity a Format Template

-

To check the validity of a format template, simply go to the Manage Format Templates page, and look at the column 'status' for the format template you want to check. If message "Ok" is there, - then no problem was found with the template. If message 'Not Ok' is in the column, click on it to see - the problems that have been found for the template.

- -

3.12 XSL Format Templates

-

In addition to the HTML-like syntax introduced in previous sections, BibFormat also - has support for server-side XSL transformation. Although you can do all the formatting using this custom HTML syntax, there are - cases where an XSL stylesheet might be preferred. XSLT is for example a natural choice - when you need to output complex XML, especially when your XML has a deep tree structure. - You might also prefer using XSLT if you already feel comfortable with XSL syntax.

-

XSL format templates are written using regular XSL. The template file has to be placed in the same folder - as regular format template files, and its file extension must be .xsl. The XSL template - are also visible through the web interface, as any regular format template file. However, some - functions like the "Dependencies checker" or the possibility to create a template or edit its attributes are not - available for the XSL templates.

- -

In BibFormat XSL you have access to the following functions, provided you have declared xmlns:fn="http://cdsweb.cern.ch/bibformat/fn" in your stylesheet:

-
-
fn:modification_date(recID)
-
Returns the record modification date. Eg: <xsl:value-of select="fn:modification_date(445)"/> returns modification date of record 445
-
-
-
fn:creation_date(recID)
-
Returns the record creation date. Eg: <xsl:value-of select="fn:creation_date(445)"/> returns creation date of record 445
-
-
-
fn:eval_bibformat(recID, bibformat_template_code)
-
Returns the results of the evaluation of the format template code. Eg: <xsl:value-of select="fn:eval_bibformat(marc:controlfield[@tag='001'],'&lt;BFE_SERVER_INFO var=&quot;recurl&quot;>')" /> returns the url of the current record. The parameter bibformat_template_code is regular code used inside BibFormat format templates, with < escaped as &lt; and "(quotes) escaped as &quot;
-
-

Finally, please note that you will need to install a supported XSLT parser in order - to format using XSL stylesheets.

- -

4. Format Elements

-

Format elements are the bricks used in format templates to provide dynamic content to the formatting process. - Their purpose is to allow non computer literate persons to easily integrate data from the records in the database into their templates.

-

Format elements are typically written in Python (there is an exception to that point which is dicussed in Add a Format Element). This brings great flexibily and power to the formatting process. This however restricts the creation of format elements to developers.

- -

4.1 Add a Format Element

-

The most typical way of adding a format element is to drop a .py file in the lib/python/invenio/bibformat_elements directory of your Invenio installation. See Edit the Code of a Format Element to learn how to implement an element.

-

The most simple way to add a format element is to add a en entry in the "Logical Fields" management interface of the BibIndex module. When BibFormat cannot find the Python format element corresponding to a given name, it looks into this table for the name and prints the value of the field declared for this name. This lightweight way of doing is straightforward but does not allow complex handling of the data (it limits to printing the value of the field, or the values of the fields if multiple fields are declared under the same label).

-

4.2 Remove a Format Element

-

To remove a Python format element simply remove the corresponding file from the lib/python/invenio/bibformat_elements directory of your Invenio installation.

-

To remove a format element declared in the "Logical Fields" management interface of the BibIndex module simply remove the entry from the table.

-

4.3 Edit the Code of a Format Element

-

This section only applies to Python format elements. Basic format elements declared in "Logical Fields" have non configurable behaviour.

-

A format element file is like any regular Python program. It has to implement a format_element function, which returns a string and takes at least bfo as first parameter (but can take as many others as needed).

-

Here is for example the code of the "bfe_title.py" element: -

-def format_element(bfo, separator=" "):
-    """
-    Prints the title of a record.
-
-    @param separator separator between the different titles
-    """
-    titles = []
-
-    title = bfo.field('245__a')
-    title_remainder = bfo.field('245__b')
-
-    titles.append( title + title_remainder )
-
-    title = bfo.field('246__a')
-    if len(title) > 0:
-        titles.append( title )
-
-    title = bfo.field('246_1a')
-    if len(title) > 0:
-        titles.append( title )
-
-    return separator.join(titles)
-
-In format templates this element can be called like a function, using HTML syntax:
-<BFE_TITLE separator="; "/>
-Notice that the call uses (almost) the filename of your element. To find out which element to use, BibFormat tries different filenames until the element is found: it tries to
    -
  1. ignore the letter case
    2. -
  2. replace underscore with spaces
    3. -
  3. remove the BFE_ from the name
    4. -
- This means that even if the filename of your element is "my element.py", BibFormat can resolve the call <BFE_MY_ELEMENT /> in a format template. This also means that you must take care no to have two format elements filenames that only differ in term of the above parameters. -

-

The string returned by the format_element function corresponds to the value that is printed instead of the format element name in the format template.

-

The bfo object taken as parameter by format_element function stands for BibFormatObject: it is an object that represents the context in which the formatting takes place. For example it allows to retrieve the value of a given field for the record that is being formatted, or the language of the user. We see the details of the BibFormatObject further below.

-

The format_element function of an element can take other parameters, as well as default values for these parameters. The idea is that these parameters are accessible from the format template when calling the elements, and allow to parametrize the behaviour of the format element.

-

It is very important to document your element: this allows to generate a documentation for the elements accessible to people writing format templates. It is the only way for them to know what your element do. The key points are: -

    -
  • Provide a docstring for the format_element function
    • -
  • For each of the parameters of the format_element function (except for bfo) as provide a description using a Java-like doc syntax in the doc string:
    @param my_param: description for my param (one line per parameter)
    • -
  • You can use one @see followed by a comma separated list of elements filenames to provide a reference to other elements of interests related to this one:
    @see my_element1.py, my element2.py
    • -
-

-

Typically you will need to get access to some fields of a record to display as output. There are two ways to this: you can access the bfo object given as parameter and use the provided (basic) accessors, or import a dedicated module and use its advanced functionalities.

-

Method 1: Use accessors of bfo:
- bfo is an instance of the BibFormatObject class. The following methods are available: -

    -
  • get_record(): Returns the record of this BibFormatObject instance as a BibRecord structure. Allows advanced access on the structure using BibRecord.
    • -
  • control_field(tag): Returns the value of control field given by MARC tag.
    • -
  • field(tag):Returns the value of the field corresponding to MARC tag. If the value does not exist, return empty string.
    • -
  • fields(tag): Returns the list of values corresonding to MARC tag.If tag has an undefined subcode (such as 999C5), the function returns a list of dictionaries, whoose keys are the subcodes and the values are the values of tag.subcode. If the tag has a subcode, simply returns list of values corresponding to tag.
    • -
  • kb(kb, string, default=""): Returns the value of the string in the knowledge base kb. If kb does not exist or string does not exist in kb, returns default string.
    • -
-You can also get access to other information through bfo, such as the language in which the formatting should occur with bfo.lang. To learn more -about the possibilities offered by the bfo, read the BibFormat APIs -

-

Method 2: Use module BibRecord:
- - BibRecord is a module that provides advanced functionalities - regarding access to the field of a record - bfo.get_record() returns a structure that can be - understood by BibRecord's functions. Therefore you can import - the module's functions to get access to the fields you want. -

-

4.4 Preview a Format Element

-

- You can play with a format element parameters and see the result - of the element directly in the - format elements documentation: - for each element, under the section "See - also", click on "Test this element". You are redirected to a page - where you can enter a value for the parameters. A description is - associated with each parameter as well as an indication of the - default value of the parameter if you do not provide a custom - value. Click on the "Test!" button to see the result of the - element with your parameters.

- -

4.5 Internationalization (i18n)

- -

You can follow the standard internationalization procedure in - use accross Invenio sources. For example the following code - will get you the translation for "Welcome" (assuming "Welcome" has - been translated): -

-from invenio.base.i18n import gettext_set_language
-
-ln = bfo.ln
-_ = gettext_set_language(ln)
-
-translated_welcome =  _("Welcome")
-
-

- -

Notice the access to bfo.ln to get access to the - current language of the user. For simpler translations or - behaviour depending on the language you can simply check the value - bfo.ln to return your custom text.

- -

4.6 Escaping special HTML/XML characters

-

In most cases, that is cases where your - element does not return HTML output, you do not have to take any - particular action in order to escape values that you output: the - BibFormat engine will take care of escaping the returned value of the element - for you. In cases where you want to return text that should not - be escaped (for example when you return HTML links), you can make - the formatting engine know that it should not escape your - value. This is done by implementing the - escape_values(bfo) function in your element, that - will return (int) 0 when escape should not be done (or 1 when - escaping should be done): - - -

def escape_values(bfo):
-    """
-    Called by BibFormat in order to check if output of this element
-    should be escaped.
-    """
-    return 0
-
- - Note that the function is given a bfo object as - parameter, such that you can do additional testing if your element - should really return 1 or 0 (for very special cases).
Also note - that the behavior defined by the escape_values() function - will be overriden by the escape parameter used in the - format template if it is specified. -

- -

Finally, be cautious when you disable escaping: you will have - to take care of escaping values "manually" in your format element - code, in order to avoid non valid outputs or XSS - vulnerabilities. This can be done easily when using the - field, fields and - controlfield functions of bfo with escape - parameter: - -

-    title = bfo.field('245__a', escape="1")
-    abstract = bfo.field('520__a', escape="2")
-
- - The escape parameter can be one of the following values: -
  • 0 - no escaping
    • -
  • 1 - escape all HTML characters (escaped chars are shown as escaped)
    • -
  • 2 - remove unsafe HTML tags to avoid XSS, but - keep basic one (such as <br />) - This is particularly useful if you want to store HTML text in your - metadata but still want to escape some tags to prevent - XSS vulnerabilities. Note that this method is slower than - basic escaping of mode 1.
    - Escaped tags are removed.
    • -
  • 3 - mix of mode 1 and mode 2. If field_value starts with <!--HTML-->, - then use mode 2. Else use mode 1.
    • -
  • 4 - remove all HTML/XML tags
    • -
  • 5 - same as 2, but allows more tags, like <img>
    • -
  • 6 - same as 3, but allows more tags, like <img>
    • -
  • 7 - mix of mode 0 and mode 1. If field_value starts with <!--HTML-->, - then use mode mode 0. Else use mode 1.
    • -
  • 8 - same as mode 1, but also escape double-quotes
    • -
  • 9 - same as mode 4, but also escape double-quotes
    • -
- These modes are the same for escape_values(bfo) function. -

- You can also decide not to use the escape parameter and escape values - using any other Python function/library you want to use (such as cgi.escape()). -

- -

As a BibFormat element developer you can also override the - default escape parameter of your format elements: - that is especially useful if you want to provide a way for format - templates editors to call your element with a custom escaping - mode that should not escape the whole output of your - element. The bfe_abstract.py element is an example - of code that overrides the escape parameter.

- -

4.7 Edit the Attributes of a Format Element

-

A format element has mainly four kinds of attributes:

    -
  • Name: it corresponds to the filename of the element.
    • -
  • Description: the description is in the docstring of the format_element function (excepted lines prefixed with @param and @see).
    • -
  • Parameters descriptions: for each parameter of the format_element function, a line beginning with @param parameter_name and followed by the description of the parameter is present in the docstring of the format_element function.
    • -
  • Reference to other elements: one line beginning with @see and followed by a list of comma-separated format elements filenames in the in the docstring of the format_element function provides a link to related elements.
    • -
-

-

4.8 Check the Dependencies of a Format Element

-

There are two ways to check the dependencies of a format element. The simplest way is to go to the format elements documentation and click on "Dependencies of this element" for the element you want to check.

-

The second method to check the dependencies of an element is through regular unix tools: for example $ grep -r -i 'bfe_your_element_name' . inside the format templates directory will tell you which templates call your element.

-

4.9 Check the Validity of a Format Element

-

There are two ways to check the validity of an element. The simplest one is to go to the format elements documentation and click on "Correctness of this element" for the element you want to check.

-

The second method to check the validity of an element is through regular Python methods: you can for example import the element in the interactive interpreter and feed it with test parameters. Notice that you will need to build a BibFormatObject instance to pass as bfo parameter to the format_element function of your element.

-

4.10 Browse the Format Elements Documentation

-

Go to the format elements documentation. There is a summary of all available format elements at the top of the page. You can click on an element to go to its detailed description in the second part of the page.

-

Each detailed documentation shows you: -

    -
  • A description of what the element does.
    • -
  • A list of all parameters you can use for this element.
    • -
  • For each parameter, a description and the default value when parameter is ommitted.
    • -
  • A link to a tool to track the dependencies of your element.
    • -
  • A link to a tool to check the correctness of your element.
    • -
  • A link to a tool to test your element with custom parameters.
    • - -

- -

5. Run BibReformat

-

While records can be formatted on-the-fly using BibFormat, it is usually necessary to preformat the records -in order to decrease the load of your server. To do so, use the bibreformat command line tool.

-

5.1 Run BibReformat

-

The following options are available for running bibreformat:

-

-
 Usage: bibreformat [options]
- -u, --user=USER         User name to submit the task as, password needed.
- -h, --help              Print this help.
- -V, --version           Print version information.
- -v, --verbose=LEVEL     Verbose level (0=min,1=normal,9=max).
- -s, --sleeptime=SLEEP   Time after which to repeat tasks (no)
- -t, --time=DATE         Moment for the task to be active (now).
- -a, --all               All records
- -c, --collection        Select records by collection
- -f, --field             Select records by field.
- -p, --pattern           Select records by pattern.
- -o, --format            Specify output format to be (re-)created. (default HB and RECJSON)
- -n, --noprocess         Count records to be processed only (no processing done)
- Example: bibreformat -n Show how many records are to be bibreformated.
-
-
-For example, to reformat all records in HB (=HTML brief) and RECJSON formats, you'd launch: -
-
$ bibreformat -a -oHB,RECJSON
-
-
-and you watch the progress of the process via bibsched. -

Note that BibReformat understands -p, -f, -and -c arguments that enable you to easily reformat only -the records you need. For example, to reformat the Pictures -collection, launch: - -

-

-

$ bibreformat -cPictures -oHB
-
-
-or to reformat HD (=HTML detailed) format for records #10 to #20, you -launch: -
-
$ bibreformat -p"recid:10->20" -oHD
-
-
-

Last but not least, if you launch bibreformat without arguments: -

-
$ bibreformat
-
-
-

-it will process all the records that have been modified since the last -run of BibReformat, as well as all newly inputted records. This is -suitable for running BibReformat in a periodical daemon mode via -BibSched. See our HOWTO Run -Your Invenio Installation guide for more information.

- -

6. Appendix

-

6.1 MARC Notation in Formats

-

The notation for accessing fields of a record are quite flexible. You can use a syntax strict regarding MARC 21, but also - a shortcut syntax, or a syntax that can have a special meaning.

-

The MARC syntax is the following one: - tag[indicator1][indicator2] [$ subfield] where tag is 3 digits, indicator1 and indicator2 are 1 character each, and subfield is 1 letter. -

-

For example to get access to an abstract you can use the MARC notation 520 $a. You can use this syntax in BibFormat. However you can also: -

    -
  • Omit any whitespace character (or use as many as you want)
    • -
  • Omit the $ character (or use as many as you want)
    • -
  • Omit or use both indicators. You cannot specify only one indicator. If you need to use only one, use underscore _ character for the other indicator.
    • -
  • Use percent '%' instead of any character to specify all ("don't care" or wildcard character) for that character.
    • -

-

6.2 Migrating from Previous BibFormat

-

The new Python BibFormat formats are not backward compatible with the previous formats. New concepts and capabilities have been introduced and some have been dropped. If you have not modified the "Formats" or modified only a -little bit the "Behaviours" (or modified "Knowledge Bases"), then the transition will be painless and -automatic. Otherwise you will have to manually rewrite some of the -formats. This should however not be a big problem.

-

The first thing you should do is to read the Five Minutes Introduction to BibFormat to understand how the new BibFormat works. We also assume that you are familiar with the concepts of the old BibFormat. As the new formats separate the presentation from the business logic (i.e. the bindings to the database), it is not possible to automatically handle the translation. This is why you should at least be able to read and understand the formats that you want to migrate.

- -

Differences between old and new BibFormat

-

-The most noticeable differences are:
-
- a) "Behaviours" have been renamed "Output formats".
- b) "Formats" have been renamed "Format templates". They are now - written in HTML.
- c) "User defined functions" have been dropped.
- d) "Extraction rules" have been dropped.
- e) "Link rules" have been dropped.
- f) "File formats" have been dropped.
- g) "Format elements" have been introduced. They are written in Python, - and can simulate c), d) and e).
- h) Formats can be managed through web interface or through - human-readable config files.
- i) Introduction of tools like validator and dependencies checker.
- j) Better support for multi-language formatting.
-

-

-Some of the advantages are:
-
- + Management of formats is much clearer and easier (less concepts, - more tools).
- + Writing formats is easier to learn : less concepts - to learn, redesigned work-flow, use of existing well known and - well documented languages.
- + Editing formats is easier: You can use your preferred HTML editor such as - Emacs, Dreamweaver or Frontpage to modify templates, or any text - editor for output formats and format elements. You can also use the - simplified web administration interface.
- + Faster and more powerful templating system.
- + Separation of business logic (output formats, format elements) - and presentation layer (format templates). This makes the management - of formats simpler.
-

-

-The disadvantages are:
-
- - No backward compatibility with old formats.
- - Stricter separation of business logic and presentation layer:
- no more use of statements such as if(), forall() inside templates, - and this requires more work to put logic inside format elements.
-

-

Migrating behaviours to output formats

-

Behaviours were previously stored in the database and did require to use the evaluation language to -provide the logic that choose which format to use for a record. They also let you enrich records -with some custom data. Now their use has been simplified and rectricted to equivalence tests on the value of a field -of the record to define the format template to use.

-

For example, the following behaviour:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CONDITIONS
0$980.a="PICTURE"
Action (0)"<record> -
 <controlfield tag=\"001\">" $001 "</controlfield> -
 <datafield tag=\"FMT\" ind1=\"\" ind2=\"\">  - -
 <subfield code=\"f\">hb</subfield>  -
 <subfield code=\"g\">"  -
xml_text(format("PICTURE_HTML_BRIEF")) -
" </subfield>  -
 </datafield> -
</record>"
  - -
100""=""
Action (0)"<record> -
 <controlfield tag=\"001\">" $001 "</controlfield> - -
 <datafield tag=\"FMT\" ind1=\"\" ind2=\"\">  -
 <subfield code=\"f\">hb</subfield>  -
 <subfield code=\"g\">"  -
xml_text(format("DEFAULT_HTML_BRIEF")) -
" </subfield>  - -
 </datafield> -
</record>"
-   -
-

translates to the following output format (in textual configuration file):

-

- -tag 980__a:
-PICTURE --- Picture_HTML_brief.bft
-default: Default_HTML_brief.bft
-

-

or visual representation through web interface:
-Image representation of HB output format -

-

Migrating formats to format templates and format elements

-

The migration of formats is the most difficult part of the migration. You will need to separate the presentation code (HTML) from the business code (iterations, tests and calls to the database). Here are some tips on how you can do this:

-
    -
  • If you want to save the time of unescaping all HTML characters and understanding how the layout should look like, just go with your web browser to a formatted version of the format in your Invenio installation, and copy the source of the web page. Identify the parts of the HTML code which are specific to the current record, and replace them with a call to the corresponding format element.
    • -
  • If you have made small modifications to the old default provided formats, we suggest that you use the new provided ones and modify them according to your needs.
    • -
-

Migrating UDFs and Link rules

-

User Defined Functions and Link rules have been dropped in the new BibFormat. These concepts have no reasons to be as they can be fully implemented in the format elements. For example the AUTHOR_SEARCH link rule can directly be implemented in the Authors.bfe element.

-

As for the UDFs, most of them are directly built-in functions of Python. Whenever a special function as to be implemented, it can be defined in a regular Python file and used in any element.

-

The Migration Kit

-

The migration kit is only available in older versions of Invenio.
-To enable the migration kit in this release, you need to copy the required files from an older release. -

-

Run old and new formats side by side

-

This was possible by default only in older versions of Invenio.
-To enable this functionality in this release, you need to copy the required files from an older release. -

-

6.3 Integrating BibFormat into Dreamweaver MX

-

BibFormat templates have been thought to be editable in custom HTML editors. We propose in this section -a way to extend one particular editor, Dreamweaver.

-

Make Dreamweaver Recognize Format Elements in Layout View

-

To make Dreamweaver understand the format elements and display an icon for each of them in the layout editor, you must -edit a Dreamweaver configuration file named Tags.xml located inside /Configuration/ThirdPartyTags directory -of your Dreamweaver installation folder. At the end of this file, copy-paste the following lines: - 

-  <!-- BibFormat (Invenio) -->
-  <tagspec tag_name="BIBFORMAT" start_string="<BFE_" end_string="/>" parse_attributes="false" detect_in_attribute="true" icon="bibformat.gif" icon_width="25" icon_height="16"></tagspec >
-  <tagspec tag_name="BIBFORMAT" start_string="<bfe_" end_string="/>" parse_attributes="false" detect_in_attribute="true" icon="bibformat.gif" icon_width="25" icon_height="16"></tagspec >
-
- Also copy this icon bibformat.gif in the same directory as Tags.xml (right-click on icon, or ctrl-click on one-button mouse, and "Save Image As..."). Make sure the downloaded image is named "bibformat.gif". -

-

Note that Dreamweaver might not recognize Format Elements when complex formatting is involved due to these elements.

-

Add a Format Elements Floating Panel

-

You can add a floating panel that will you to insert Format Elements in your document and read the documentation - of all available Format Elements.

-

The first step is to declare in which menu of Dreamweaver this floating panel is going to be available. - To do so, edit file "Menu.xml" located inside /Configuration/Menus of your Dreamweaver - application directory and copy-paste the following line in the menu you want - (typically inside tag 'menu' with attribute id='DWMenu_Window_Others'):

- 
-   <menuitem name="BibFormat Elements" enabled="true" command="dw.toggleFloater('BibFormat_floater.html')" checked="dw.getFloaterVisibility('BibFormat_floater.html')" />
-
-

-

Once this is done, you can download the floating palette (if file opens in your browser instead of downloading, right-click on icon, or ctrl-click on one-button mouse, and "Save Target As...") and move the dowloaded file "BibFormat_floater.html" (do not rename it) into /Configuration/Floaters directory of your Dreamweaver application folder.

-

To use the BibFormat floating panel, open Dreamweaver, and choose Window > Others > BibFormat Elements.

-

Whenever a new version of the palette is available, you can skip the edition of file "Menu.xml" and just replace the old "BibFormat_floater" file with the new one.

-

6.4 FAQ

- -

Why do we need output formats? Wouldn't format templates be sufficient?

-

As you potentially have a lot of records, it is not conceivable to specify for each of them which -format template they should use. This is why this rule-based decision layer has been introduced.

- -

How can I protect a format?

-

As a web user, you cannot protect a format. If you are administrator of the -system and have access to the format files, you can simply use the permission rights of your system, as BibFormat -is aware of it.

- -

Why cannot I edit/delete a format?

-

The format file has certainly been protected by the administrator of the server. You must ask the -administrator to unprotect the file if you want to edit it.

- -

How can I add a format element from the web interface?

-

Format elements cannot be added, removed or edited through the web interface. This limitation -has been introduced to limit the security risks caused by the upload of Pythonic files on the server. The only possibility to add a basic format element from the web interface is to add a en entry in the "Logical Fields" management interface of the BibIndex module (see Add a Format Element)

- -

Why are some Marc codes omitted in the "Check Dependencies" pages?

-

When you check the dependencies of a format, the page reminds you that -some use of Marc codes might not be indicated. This is because it is not -possible (or at least not trivial) to guess that the call to field(str(5+4)+"80"+"__a") -is equal to a call to field("980__a"). You should then not completely rely on this indication.

- -

How are displayed deleted record?

-

By default, Invenio displays a standard "The record has been deleted." message for all -output formats with a 'text/html' content type. Your output format, format templates and format elements -are bypassed by the engine. -However, for more advanced output formats, Invenio -goes through the regular formatting process and let your formats do the job. This allows you to customize how a record should be displayed once it has been deleted.

- -

Why are some format elements omitted in the "Knowledge Base Dependencies" page?

-

When you check the dependencies of a knowledge base, the page -reminds you that format elements using this knowledge base might not -be indicated. This is because it is not possible (or at least not -trivial) to guess that the call to -kb(e.upper()+"journal"+"s") in a format element is equal -to a call to kb("Ejournals"). You should then not -completely rely on this indication.

- -

Why are some format elements defined in field table omitted in the format element documentation?

- -

Some format elements defined in the "Logical Fields" management -interface of the BibIndex module (the basic format elements) are not -shown in the format elements documentation pages. We do not show such -an element if its name starts with a number. This is to reduce the -number of elements shown in the documentation as the logical fields -table contains a lot of not so useful fields to be used in -templates.

- -

How can I get access to repeatable subfields from inside a format element?

- -

Given that repeatable subfields are not frequent, the -bfo.fields(..) function has been implemented to return the most convenient structure for most cases, that is a 'list of strings' (Case 1 below) or 'list of dict of strings' (Case 2 below). For eg. with the following metadata: -

    999C5 $a value_1a $b value_1b
-    999C5 $b value_2b
-    999C5 $b value_3b $b value_3b_bis
-
-    >> bfo.fields('999C5b')                                   (1)
-    >> ['value_1b', 'value_2b', 'value_3b', 'value_3b_bis']
-    >> bfo.fields('999C5')                                    (2)
-    >> [{'a':'value_1a', 'b':'value_1b'},
-        {'b':'value_2b'},
-        {'b':'value_3b'}]
-
- -In this example value3b_bis is not shown for -bfo.fields('999C5') (Case 2). If it were to be taken into account, the -returned structure would have to be a 'list of dict of list of strings', thus making for most cases -the access to the data a bit more complex.
-In order to consider the repeatable subfields, use the additional repeatable_subfields_p parameter: - -
    >> bfo.fields('999C5b', repeatable_subfields_p=True)      (1 bis)
-    >> ['value_1b', 'value_2b', 'value_3b']
-    >> bfo.fields('999C5', repeatable_subfields_p=True)       (2 bis)
-    >> [{'a':['value_1a'], 'b':['value_1b']},
-        {'b':['value_2b']},
-        {'b':['value_3b', 'value3b_bis']}]
-
-Another solution would be to access the BibRecord structure with -bfo.getRecord() and use the lower-level BibRecord module with this structure. -

-
diff --git a/invenio/legacy/bibformat/doc/hacking/bibformat-api.webdoc b/invenio/legacy/bibformat/doc/hacking/bibformat-api.webdoc deleted file mode 100644 index 47f3efc0f..000000000 --- a/invenio/legacy/bibformat/doc/hacking/bibformat-api.webdoc +++ /dev/null @@ -1,877 +0,0 @@ -# -*- mode: html; coding: utf-8; -*- - -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - - - - - - -
-****************************************************************************
-** IMPORTANT NOTE: Note that this documentation is an updated version of  **
-** an earlier technical draft of BibFormat specifications. Please first   **
-** refer to the BibFormat admin guide.                                    **
-****************************************************************************
-
-Technical Overview of the new BibFormat
-=======================================
-
-Contents:
-1. Python API
-2. The philosophy behind BibFormat
-3. Differences between the old PHP version and the new Pythonic version
-4. Migrating from the previous PHP BibFormat version to the new Pythonic version
-5. Specifications of the new BibFormat configuration files.
-
-
-1. Python API
-
-The APIs of bibformat.py consists in these functions:
-
- def format_record(recID, of, ln=CFG_SITE_LANG, verbose=0,
-                   search_pattern=None, xml_record=None, user_info=None,
-                   on_the_fly=False):
-     """
-     Formats a record given its ID (or its XML representation)
-     and an output format.
-
-     Returns a formatted version of the record in the specified
-     language, with pattern context, and specified output format.
-     The function will define by itself which format template must be
-     applied.
-
-     Parameters that allow contextual formatting (like 'search_pattern'
-     and 'user_info') are useful only when doing on-the-fly
-     formatting, or when caching with care (e.g. caching all formatted
-     versions of a record for each possible 'ln').
-
-     The arguments are as follows:
-
-               recID -  the ID of the record to format. If ID does not exist
-                        the function returns empty string or an error
-                        string, depending on level of verbosity.
-                        If 'xml_record' parameter is specified, 'recID'
-                        is ignored
-
-                  of -  an output format code. If 'of' does not exist as code in
-                        output format, the function returns empty
-                        string or an error string, depending on level
-                        of verbosity.  ;of' is case insensitive.
-
-                  ln -  the language to use to format the record. If
-                        'ln' is an unknown language, or translation
-                        does not exist, default CFG_SITE_LANG language
-                        will be applied whenever possible.
-                        Allows contextual formatting.
-
-             verbose -  the level of verbosity in case of errors/warnings
-                        0 - Silent mode
-                        5 - Prints only errors
-                        9 - Prints errors and warnings
-
-      search_pattern -  the pattern used as search query when asked to
-                        format this record (User request in web
-                        interface). Allows contextual formatting.
-
-          xml_record -  an XML string representation of the record to
-                        format.  If it is specified, recID parameter is
-                        ignored. The XML must be pasable by BibRecord.
-
-           user_info - allows to grant access to some functionalities
-                       on a page depending on the user's
-                       priviledges. 'user_info' is the same structure
-                       as the one returned by webuser.collect_user_info(req),
-                       (that is a dictionary).
-
-          on_the_fly - if False, try to return an already preformatted
-                       version of the record in the database.
-
-     """
-
-
- Example:
-   >> from invenio.modules.formatter import format_record
-   >> format_record(5, "hb", "fr")
-
-
- def format_records(recIDs, of, ln=CFG_SITE_LANG, verbose=0, search_pattern=None,
-                    xml_records=None, user_info=None, record_prefix=None,
-                    record_separator=None, record_suffix=None,
-                    prologue="", epilogue="", req=None, on_the_fly=False):
-     """
-     Returns a list of formatted records given by a list of record IDs or a
-     list of records as xml.
-     Adds a prefix before each record, a suffix after each record,
-     plus a separator between records.
-
-     Also add optional prologue and epilogue to the complete formatted list.
-
-     You can either specify a list of record IDs to format, or a list of
-     xml records, but not both (if both are specified recIDs is ignored).
-
-     'record_separator' is a function that returns a string as separator between
-     records. The function must take an integer as unique parameter,
-     which is the index in recIDs (or xml_records) of the record that has
-     just been formatted. For example separator(i) must return the separator
-     between recID[i] and recID[i+1]. Alternatively separator can be a single
-     string, which will be used to separate all formatted records.
-     The same applies to 'record_prefix' and 'record_suffix'.
-
-     'req' is an optional parameter on which the result of the function
-     are printed lively (prints records after records) if it is given.
-     Note that you should set 'req' content-type by yourself, and send
-     http header before calling this function as it will not do it.
-
-     This function takes the same parameters as 'format_record' except for:
-
-               recIDs -  a list of record IDs to format
-
-          xml_records -  a list of xml string representions of the records to
-                         format. If this list is specified, 'recIDs' is ignored.
-
-        record_prefix - a string or a function the takes the index of the record
-                        in 'recIDs' or 'xml_records' for which the function must
-                        return a string.
-                        Printed before each formatted record.
-
-     record_separator - either a string or a function that returns string to
-                        separate formatted records. The function takes the index
-                        of the record in 'recIDs' or 'xml_records' that is being
-                        formatted.
-
-        record_prefix - a string or a function the takes the index of the record
-                        in 'recIDs' or 'xml_records' for which the function must
-                        return a string.
-                        Printed after each formatted record
-
-                  req - an optional request object on which formatted records
-                        can be printed (for "live" output )
-
-             prologue - a string printed before all formatted records string
-
-             epilogue - a string printed after all formatted records string
-
-           on_the_fly - if False, try to return an already preformatted version
-                        of the records in the database
-     """
-
-
- def get_output_format_content_type(of):
-     """
-     Returns the content type (eg. 'text/html' or 'application/ms-excel') \
-     of the given output format.
-
-     The function takes this mandatory parameter:
-
-     of - the code of output format for which we want to get the content type
-     """
-
-
-The API of the BibFormat Object ('bfo') given as a parameter to
-format function of format elements consist in the following
-functions. This API is to be used only inside format elements.
-
- def control_field(self, tag, escape='0'):
-    """
-    Returns the value of control field given by tag in record.
-
-    If the value does not exist, returns empty string
-    The returned value is always a string.
-
-    'escape' parameter allows to escape special characters
-    of the field. The value of escape can be:
-          0 - no escaping
-          1 - escape all HTML characters
-          2 - remove unsafe HTML tags (Eg. keep <br />)
-          3 - Mix of mode 1 and 2. If value of field starts with
-              <!-- HTML -->, then use mode 2. Else use mode 1.
-          4 - Remove all HTML tags
-          5 - Same as 2, with more tags allowed (like <img>)
-          6 - Same as 3, with more tags allowed (like <img>)
-          7 - Mix of mode 0 and mode 1. If field_value starts
-              with <!-- HTML -->, then use mode 0. Else use mode 1.
-          8. Same as mode 1, but also escape double-quotes
-          9. Same as mode 4, but also escape double-quotes
-
-    The arguments are:
-
-         tag    -  the marc code of a field
-         escape -  1 if returned value should be escaped. Else 0.
-                   (see above for other modes)
-    """
-
- def field(self, tag, escape='0'):
-    """
-    Returns the value of the field corresponding to tag in the
-    current record.
-
-    If the value does not exist, returns empty string
-    Else returns the same as bfo.fields(..)[0] (see docstring below).
-
-    'escape' parameter allows to escape special characters
-    of the field. The value of escape can be:
-          0 - no escaping
-          1 - escape all HTML characters
-          2 - remove unsafe HTML tags (Eg. keep <br />)
-          3 - Mix of mode 1 and 2. If value of field starts with
-              <!-- HTML -->, then use mode 2. Else use mode 1.
-          4 - Remove all HTML tags
-          5 - Same as 2, with more tags allowed (like <img>)
-          6 - Same as 3, with more tags allowed (like <img>)
-          7 - Mix of mode 0 and mode 1. If field_value starts
-              with <!-- HTML -->, then use mode 0. Else use mode 1.
-          8 - Same as mode 1, but also escape double-quotes
-          9 - Same as mode 4, but also escape double-quotes
-
-    The arguments are:
-
-         tag  -  the marc code of a field
-         escape -  1 if returned value should be escaped. Else 0.
-                   (see above for other modes)
-    """
-
-
- def fields(self, tag, escape='0', repeatable_subfields_p=False):
-    """
-    Returns the list of values corresonding to "tag".
-
-    If tag has an undefined subcode (such as 999C5),
-    the function returns a list of dictionaries, whoose keys
-    are the subcodes and the values are the values of tag.subcode.
-    If the tag has a subcode, simply returns list of values
-    corresponding to tag.
-    Eg. for given MARC:
-        999C5 $a value_1a $b value_1b
-        999C5 $b value_2b
-        999C5 $b value_3b $b value_3b_bis
-
-        >> bfo.fields('999C5b')
-        >> ['value_1b', 'value_2b', 'value_3b', 'value_3b_bis']
-        >> bfo.fields('999C5')
-        >> [{'a':'value_1a', 'b':'value_1b'},
-            {'b':'value_2b'},
-            {'b':'value_3b'}]
-    By default the function returns only one value for each
-    subfield (that is it considers that repeatable subfields are
-    not allowed). It is why in the above example 'value3b_bis' is
-    not shown for bfo.fields('999C5').  (Note that it is not
-    defined which of value_3b or value_3b_bis is returned).  This
-    is to simplify the use of the function, as most of the time
-    subfields are not repeatable (in that way we get a string
-    instead of a list).  You can allow repeatable subfields by
-    setting 'repeatable_subfields_p' parameter to True. In
-    this mode, the above example would return:
-        >> bfo.fields('999C5b', repeatable_subfields_p=True)
-        >> ['value_1b', 'value_2b', 'value_3b']
-        >> bfo.fields('999C5', repeatable_subfields_p=True)
-        >> [{'a':['value_1a'], 'b':['value_1b']},
-            {'b':['value_2b']},
-            {'b':['value_3b', 'value3b_bis']}]
-    NOTICE THAT THE RETURNED STRUCTURE IS DIFFERENT.  Also note
-    that whatever the value of 'repeatable_subfields_p' is,
-    bfo.fields('999C5b') always show all fields, even repeatable
-    ones. This is because the parameter has no impact on the
-    returned structure (it is always a list).
-
-    'escape' parameter allows to escape special characters
-     of the fields. The value of escape can be:
-                  0 - no escaping
-                  1 - escape all HTML characters
-                  2 - remove unsafe HTML tags (Eg. keep <br />)
-                  3 - Mix of mode 1 and 2. If value of field starts with
-                      <!-- HTML -->, then use mode 2. Else use mode 1.
-                  4 - Remove all HTML tags
-                  5 - Same as 2, with more tags allowed (like <img>)
-                  6 - Same as 3, with more tags allowed (like <img>)
-                  7 - Mix of mode 0 and mode 1. If field_value starts
-                      with <!-- HTML -->, then use mode 0. Else use mode 1.
-                  8 - Same as mode 1, but also escape double-quotes
-                  9 - Same as mode 4, but also escape double-quotes
-
-    The arguments are:
-
-          tag  -  the marc code of a field
-          escape -  1 if returned value should be escaped. Else 0.
-                   (see above for other modes)
-          repeatable_subfields_p - if True, returns the list of
-                                   subfields in the dictionary @return
-                                   values of field tag in record """
-
- def kb(self, kb, string, default=""):
-    """
-    Returns the value of the "string" in the knowledge base "kb".
-
-    If kb does not exist or string does not exist in kb,
-    returns 'default' string or empty string if not specified
-
-    The arguments are as follows:
-
-          kb  -  the knowledge base name in which we want to find the mapping.
-                 If it does not exist the function returns the original
-                 'string' parameter value. The name is case insensitive (Uses
-                 the SQL 'LIKE' syntax to retrieve value).
-
-      string  -  the value for which we want to find a translation-
-                 If it does not exist the function returns 'default' string.
-                 The string is case insensitive (Uses the SQL 'LIKE' syntax
-                 to retrieve value).
-
-     default  -  a default value returned if 'string' not found in 'kb'.
-
-    """
-
- def get_record(self):
-    """
-    Returns the record encapsulated in bfo as a BibRecord structure.
-    You can get full access to the record through bibrecord.py functions.
-    """
-
-  Example (from inside BibFormat element):
-  >> bfo.field("520.a")
-  >> 'We present a quantitative appraisal of the physics potential
-      for neutrino experiments.'
-  >>
-  >> bfo.control_field("001")
-  >> '12'
-  >>
-  >> bfo.fields("700.a")
-  >>['Alekhin, S I', 'Anselmino, M', 'Ball, R D', 'Boglione, M']
-  >>
-  >> bfo.kb("DBCOLLID2COLL", "ARTICLE")
-  >> 'Published Article'
-  >>
-  >> bfo.kb("DBCOLLID2COLL", "not in kb", "My Value")
-  >> 'My Value'
-
-Moreover you can have access to the language requested for the
-formatting, the search pattern used by the user in the web
-interface and the userID by directly getting the attribute from 'bfo':
-
-    bfo.lang
-    """
-    Returns the language that was asked to be used for the
-    formatting. Always returns a string.
-    """
-
-    bfo.search_pattern
-    """
-    Returns the search pattern specified by the user when
-    the record had to be formatted. Always returns a string.
-    """
-
-    bfo.user_info
-    """
-    Returns a dictionary with information about current user.
-    The returned dictionary has the following structure:
-        user_info = {
-            'remote_ip' : '',
-            'remote_host' : '',
-            'referer' : '',
-            'uri' : '',
-            'agent' : '',
-            'uid' : -1,
-            'nickname' : '',
-            'email' : '',
-            'group' : [],
-            'guest' : '1'
-        }
-    """
-
-    bfo.uid
-    """
-    ! DEPRECATED: use bfo.user_info['uid'] instead
-    """
-
-    bfo.recID
-    """
-    Returns the id of the record
-    """
-
-    bfo.req
-    """
-    ! DEPRECATED: use bfo.user_info instead
-    """
-
-    bfo.format
-    """
-    ! DEPRECATED: use bfo.output_format instead
-    """
-
-    bfo.output_format
-    """
-    Returns the format in which the record is being formatted
-    """
-
-  Example (from inside BibFormat element):
-  >> bfo.lang
-  >> 'en'
-  >>
-  >> bfo.search_pattern
-  >> 'mangano and neutrino and factory'
-
-
-2. The philosophy behind BibFormat
-
-BibFormat is in charge of formatting the bibliographic records that
-are displayed to your users. As you potentially have a huge amount of
-bibliographic records, you cannot specify manually for each of them
-how it should be formatted. This is why you can define rules that will
-allow BibFormat to understand which kind of formatting to apply to a given
-record. You define this set of rules in what is called an "output
-format".
-
-You can have different output formats, each with its own characteristics.
-For example you certainly want that when multiple bibliographic records are
-displayed at the same time (as it happens in search results), only
-short versions are shown to the user, while a detailed record is
-preferable when a single record is displayed. You might also want to
-let your users decide which kind of output they want. For example you
-might need to display HTML for regular web browsing, but would also
-give a BibTeX version of the bibliographic reference for direct
-inclusion in a LaTeX document.
-See section 5.1 to learn how to create or modify output formats.
-
-While output formats define what kind of formatting must be applied,
-they do not define HOW the formatting is done. This is the role of the
-"format templates", which define the layout and look of a
-bibliographic reference. These format templates are rather easy to
-write if you know a little bit of HTML (see section 5.2 "Format
-templates specifications"). You will certainly have to create
-different format templates, for different kinds of records. For
-example you might want records that contain pictures to display them,
-maybe with captions, while records that do not have pictures limited
-to printing a title and an abstract.
-
-In summary, you have different output formats (like 'brief HTML',
-'detailed HTML' or 'BibTeX') that call different format templates
-according to some criteria.
-
-There is still one kind of configuration file that we have not talked
-about: the "format elements". These are the "bricks" that you use in
-format templates, to get the values of a record. You will learn to use
-them in your format template in section 5.2 "Format templates
-specifications", but you will mostly not need to modify them or create
-new ones. However if you do need to edit one, read section 5.3 "Format
-elements specifications" (And if you know Python it will be easy, as
-they are written in Python).
-
-Finally BibFormat can make use of mapping tables called "knowledge
-bases". Their primary use is to act like a translation table, to
-normalize records before displaying them. For example, you can say
-that records that have value "Phys Rev D" or "Physical Review D" for
-field "published in" must display "Phys Rev : D." to users. See
-section 5.4 to learn how to edit knowledge bases.
-
-In summary, there are three layers.  Output formats:
-
-+-----------------------------------------------------+
-|                    Output Format                    | (Layer 1)
-|                    eg: HTML_Brief.bfo               |
-+-----------------------------------------------------+
-
-call one of several `format templates':
-
-+-------------------------+ +-------------------------+
-|     Format Template     | |     Format Template     | (Layer 2)
-|     eg: preprint.bft    | |     eg: default.bft     |
-+-------------------------+ +-------------------------+
-
-that use one or several format elements:
-
-+--------------+ +----------------+ +-----------------+
-|Format Element| |Format Element  | | Format Element  | (Layer 3)
-|eg: authors.py| |eg: abstract.py | | eg: title.py    |
-+--------------+ +----------------+ +-----------------+
-
-
-3. Differences between the old PHP version and the new Pythonic version
-
-The most noticeable differences are:
-
- a) "Behaviours" have been renamed "Output formats".
- b) "Formats" have been renamed "Format templates". They are now
-     written in HTML.
- c) "User defined functions" have been dropped.
- d) "Extraction rules" have been dropped.
- e) "Link rules" have been dropped.
- f) "File formats" have been dropped.
- g) "Format elements" have been introduced. They are written in Python,
-     and can simulate c), d) and e).
- h)  Formats can be managed through web interface or through
-     human-readable config files.
- i)  Introduction of tools like validator and dependencies checker.
- j)  Better support for multi-language formatting.
-
-Some of the advantages are:
-
- + Management of formats is much clearer and easier (less concepts,
-   more tools).
- + Writing formats is easier to learn : less concepts
-   to learn, redesigned work-flow, use of existing well known and
-   well documented languages.
- + Editing formats is easier: You can use your preferred HTML editor such as
-   Emacs, Dreamweaver or Frontpage to modify templates, or any text
-   editor for output formats and format elements. You can also use the
-   simplified web administration interface.
- + Faster and more powerful templating system.
- + Separation of business logic (output formats, format elements)
-   and presentation layer (format templates). This makes the management
-   of formats simpler.
-
-The disadvantages are:
-
- - No backward compatibility with old formats.
- - Stricter separation of business logic and presentation layer:
-   no more use of statements such as if(), forall() inside templates,
-   and this requires more work to put logic inside format elements.
-
-
-4. Migrating from the previous PHP BibFormat version to the new Pythonic version
-
-Old BibFormat formats are no longer compatible with the new BibFormat
-files. If you have not modified the "Formats" or modified only a
-little bit the "Behaviours", then the transition will be painless and
-automatic. Otherwise you will have to manually rewrite some of the
-formats. This should however not be a big problem. Firstly because the
-Invenio installation will provide both versions of BibFormat for
-some time. Secondly because both BibFormat versions can run side by
-side, so that you can migrate your formats while your server still
-works with the old formats.  Thirdly because we provide a migration
-kit that can help you go through this process. Finally because the
-migration is not so difficult, and because it will be much easier for
-you to customize how BibFormat formats your bibliographic data.
-
-Concerning the migration kit it can:
- a) Effortlessly migrate your behaviours, unless they include complex
-    logic, which usually they don't.
- b) Help you migrate formats to format templates and format elements.
- c) Effortlessly migrate your knowledge bases.
-
-Point b) is the most difficult to achieve: previous formats did mix
-business logic and code for the presentation, and could use PHP
-functions. The new BibFormat separates business logic and
-presentation, and does not support PHP. The transition kit will try to
-move business logic to the format elements, and the presentation to
-the format templates. These files will be created for you, includes
-the original code and, if possible, a proposal of Python
-translation. We recommend that you do not to use the transition kit to
-translate formats, especially if you have not modified default
-formats, or only modified default formats in some limited places. You
-will get cleaner code if you write format elements and format
-templates yourself.
-
-
-5. Specifications of the new BibFormat configuration files.
-
-   BibFormat uses human readable configuration files. However (apart
-   from format elements) these files can be edited and managed through
-   a web interface.
-
-5.1 Output formats specifications
-
-Output formats specify rules that define which format template
-to use to format a record.
-While the syntax of output formats is basic, we recommend that you use
-the web interface do edit them, to be sure that you make no error.
-
-The syntax of output format is the following one. First you
-define which field code you put as the conditon for the rule.
-You suffix it with a column. Then on next lines, define the values of
-the condition, followed by --- and then the filename of the template
-to use:
-
-  tag 980.a:
-  PICTURE --- PICTURE_HTML_BRIEF.bft
-  PREPRINT --- PREPRINT_HTML_BRIEF.bft
-  PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
-
-This means that if value of field 980.a is equal to PICTURE, then we
-will use format template PICTURE_HTML_BRIEF.bft. Note that you must
-use the filename of the template, not the name. Also note that spaces
-at the end or beginning are not considered. On the following lines,
-you can either put other conditions on tag 980.a, or add another tag on
-which you want to put conditions.
-
-At the end you can add a default condition:
-
-   default: PREPRINT_HTML_BRIEF.bft
-
-which means that if no condition is matched, a format suitable for
-Preprints will be used to format the current record.
-
-The output format file could then look like this:
-
-  tag 980.a:
-  PICTURE --- PICTURE_HTML_BRIEF.bft
-  PREPRINT --- PREPRINT_HTML_BRIEF.bft
-  PUBLICATION --- PUBLICATION_HTML_BRIEF.bft
-
-  tag 8560.f:
-  .*@cern.ch --- SPECIAL_MEMBER_FORMATTING.bft
-
-  default: PREPRINT_HTML_BRIEF.bft
-
-You can add as many rules as you want. Keep in mind that they are read
-in the order they are defined, and that only first rule that
-matches will be used.
-Notice the condition on tag 8560.f: it uses a regular expression to
-match any email address that ends with @cern.ch (the regular
-expression must be understandable by Python)
-
-Some other considerations on the management of output formats:
-- Format outputs must be placed inside directory
-  /etc/bibformat/outputs/ of your Invenio installation.
-- Note that as long as you have not provided a name to an output
-  THROUGH the web interface, it will not be available as a choice
-  for your users in some parts of Invenio.
-- You should remove output formats THROUGH the web interface.
-- The format extension of output format is .bfo
-
-
-5.2 Format templates specifications
-
-Format templates are written in HTML-like syntax. You can use the
-standard HTML and CSS markup languague to do the formatting. The best
-thing to do is to create a page in your favourite editor, and once you
-are glad with it, add the dynamic part of the page, that is print the
-fields of the records. Let's say you have defined this page:
-
-  <h1>Some title</h1>
-  <p><i>Abstract: </i>Some abstract</p>
-
-Then you want that instead of "Some title" and "Some abstract", the
-value of the current record that is being displayed is used. To do so,
-you must use a format element brick. Either you know the name of the
-brick by heart, or you look for it in the elements documentation (see
-section 5.3). For example you would find there that you can print the
-title of the record by writing the HTML tag <BFE_TITLE /> in your
-format template, with parameter 'default' for a default value.
-
-  <h1><BFE_TITLE default="No Title"/></h1>
-  <p><BFE_ABSTRACT limit="1" prefix="<i>Abstract: </i>"
-  default="No abstract"/></p>
-
-Notice that <BFE_ABSTRACT /> has a parameter "limit" that <BFE_title/>
- had not ("limit" allows to limit the number of sentences of the
-abstract, according to the documentation). Note that while format
-elements might have different parameters, they always can take the the
-three following ones: "prefix" and "suffix", whose values are printed
-only if the element is not empty, and "default", which is printed only
-if element is an empty string. We have used "prefix" for the abstract,
-so that the label "<i>Abstract: </i>" is only printed if the record
-has an abstract.
-
-You should also provide these tags in all of your templates:
- -<name>a name for this template in the admin web interface</name>
- -<description>a description to be used in admin web interface for
-  this template</description>
-
-Another feature of the templates is the support for multi-languages
-outputs. You can include <lang> tags, which contain tags labeled with
-the names of the languages supported in Invenio. For example, one
-might write:
-
-  <lang><en>A record:</en><fr>Une notice:</fr></lang>
-  <h1><BFE_TITLE default="No Title"/></h1>
-  <p><BFE_ABSTRACT limit="1" prefix="<i>Abstract: </i>"
-  default="No abstract"/></p>
-
-When doing this you should at least make sure that the default
-language of your server installation is available in each <lang>
-tag. It is the one that is used if the requested language to display
-the record is not available. Note that we could also provide a
-translation in a similar way for the "No Title" default value inside
-<BFE_Title /> tag.
-
-Some other considerations on the use of elements inside templates:
- -Format elements names are not case sensitive
- -Format elements names always start with <BFE_
- -Format elements parameters can contain '<' characters,
-  and quotes different from the kind that delimit parameters (you can
-  for example have <BFE_Title default='<a href="#">No Title</a>'/> )
- -Format templates must be placed inside the directory
-  /etc/bibformat/templates/ of your Invenio installation
- -The format extension of a template is .bft
-
-Trick: you can use the <BFE_FIELD tag="245__a" /> to print the value
-of any field 245 $a in your templates.  This practice is however not
-recommended because it would necessitate to revise all format
-templates if you change meaning of the MARC code schema.
-
-5.3 Format elements specifications
-
-Format elements are the bricks used in format templates to provide the
-dynamic contents inside format templates.
-
-For the most basic format elements, you do not even need to write
-them: as long as you define `tag names' for MARC tags in the BibIndex
-Admin's Manage logical fields interface (database table tag),
-BibFormat knows which field must be printed when <BFE_tag_name/> is
-used inside a template.
-
-However for more complex processing, you will need to write a format
-element. A format element is written in Python. Therefore its file
-extension is ".py". The name you choose for the file is the one that
-will be used inside format template to call the element, so choose it
-carefully such that it is not too long, but self explanatory (you can
-prefix the filename with BFE or not, but the element will always be
-called with prefix <BFE_ inside templates).  Then you just need to
-drop the file in the lib/python/invenio/bibformat_elements/ directory
-of your Invenio installation. Inside your file you have to define
-a function named "format_element", which takes at least a "bfo"
-parameter (bfo for BibFormat Object). The function must return a
-string:
-
-  def format_element(bfo):
-      out = ""
-
-      return out
-
-You can have as many parameters as you want, as long as you make sure
-that parameter bfo is here. Let's see how to define an element that
-will print a brief title. It will take a parameter 'limit' that will
-limit the number of characters printed. We can provide some
-documentation for the elemen in the docstring of the
-function.
-
- def format_element(bfo, limit="10"):
-      """
-      Prints a short title
-
-      @param limit a limit for the number of printed characters
-      """
-
-      out = ""
-
-      return out
-
-Note that we put a default value of 10 in the 'limit' parameter.  To
-get some value of a field, we must request the 'bfo' object. For
-example we can get the value of field 245.a (field "title"):
-
- def format_element(bfo, limit="10"):
-      """
-      Prints a short title
-
-      @param limit a limit for the number of printed characters
-      """
-
-      title = bfo.field('245.a')
-
-      limit = int(limit)
-      if limit > len(title):
-          limit = len(title)
-
-      return title[:limit]
-
-As format elements are written in Python, we have decided not to give
-permission to edit elements through the web interface. Firstly for
-security reasons. Secondly because Python requires correct indentation,
-which is difficult to achieve through a web interface.
-
-You can have access to the documentation of your element through a web
-interface. This is very useful when you are writing a format template,
-to see which elements are available, what they do, which parameters they
-take, what are the default values of parameters, etc. The
-documentation is automatically extracted from format elements.
-Here follows an sample documentation generated for the element
-<BFE_TITLE />:
-
-+--------------------------------------------------------------------------------------------+
-|  TITLE                                                                                     |
-|  -----                                                                                     |
-|  <BFE_TITLE separator="..." prefix="..." suffix="..." default="..." />                  |
-|                                                                                            |
-|      Prints the title of a record.                                                         |
-|                                                                                            |
-|      Parameters:                                                                           |
-|            separator - separator between the different titles.                             |
-|            prefix - A prefix printed only if the record has a value for this element.      |
-|            suffix - A suffix printed only if the record has a value for this element.      |
-|            default - A default value printed if the record has no value for this element.  |
-|                                                                                            |
-|       See also:                                                                            |
-|            Format templates that use this element                                          |
-|            The Python code of this element                                                 |
-+--------------------------------------------------------------------------------------------+
-
-The more you provide documentation in the docstring of your elements,
-the easier it will be to write format template afterwards.
-
-Some remarks concerning format elements:
- -parameters are always string values
- -if no value is given as parameter in format the template, then the
-  value of parameter is "" (emtpy string)
- -the docstring should contain a description, followed by
-  "@param parameter: some description for parameter" for each
-  parameter (to give description for each parameter
-  in element documentation), and @see an_element.py, another_element.py
-  (to link to other elements in the documentation). Similar to JavaDoc.
- -the following names cannot be used as parameters:
-  "default", "prefix", "suffix" and escape. They can however always be
-  used in the format template for any element.
-
-Another important remark concerns the 'escaping' of output of format
-elements. In most cases, format elements output is to be used for
-HTML/XML. Therefore special characters such as < or & have to be
-'escaped', replaced by '<' and '&'. This is why all outputs
-produced by format elements are automatically escaped by BibFormat,
-unless specified otherwise.  This means that you do not have to care
-about meta-data that would break your HTML displaying or XML export
-such as a physics formula like 'a < b'. Please also note that value
-given in 'prefix', 'suffix' and 'default' parameters are not escaped,
-such that you can safely use HTML tags for these.
-
-There are always cases where the default 'escaping' behaviour of
-BibFormat is not desired. For example when you explicitely output HTML
-text, like links: you do not want to see them escaped.  The first way
-to avoid this is to modify the call to your format element in the
-format template, by setting the default 'escape' parameter to 0:
-
- 
-
-This is however inconvenient as you have to possibly need to modify a
-lot of templates. The other way of doing is to add another function to
-your format element, named 'escape':
-
- def escape_values(bfo):
-     """
-     Called by BibFormat in order to check if output of this element
-     should be escaped.
-     """
-     return 0
-
-In that way all calls to your format element will produce unescaped
-output.  You will have to take care of escaping values "manually" in
-your format element code, in order to avoid non valid outputs or XSS
-vulnerabilities. There are methods to ease the escaping in your code
-described in section 1.
-Please also note that if you use this method, your element can still
-be escaped if a call to your element from a format template
-explicitely specifies to escape value using parameter 'escape'.
-
-
-5.4 Knowledge bases specifications
-
-Knowledge bases cannot be managed through configuration files.
-You can very easily add new bases and mappings using the given web GUI.
-
- -- End of file --
-
- diff --git a/invenio/legacy/bibformat/doc/hacking/bibformat-internals.webdoc b/invenio/legacy/bibformat/doc/hacking/bibformat-internals.webdoc deleted file mode 100644 index 43b484582..000000000 --- a/invenio/legacy/bibformat/doc/hacking/bibformat-internals.webdoc +++ /dev/null @@ -1,34 +0,0 @@ -# -*- mode: html; coding: utf-8; -*- - -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - - - - - -

This page summarizes all the information suitable to dig inside -BibFormat internals.

- -
-
- -
BibFormat API
Explains how to call -the formatting engine from your Python programs, should a need be. - -
-
diff --git a/invenio/legacy/bibformat/scripts/__init__.py b/invenio/legacy/bibformat/scripts/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/invenio/legacy/bibformat/scripts/bibreformat.py b/invenio/legacy/bibformat/scripts/bibreformat.py deleted file mode 100644 index af0bcc977..000000000 --- a/invenio/legacy/bibformat/scripts/bibreformat.py +++ /dev/null @@ -1,31 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2010, 2011, 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""Call BibFormat engine and create HTML brief (and other) formats for - bibliographic records. Upload formats via BibUpload.""" - -__revision__ = "$Id$" - -from invenio.base.factory import with_app_context - - -@with_app_context() -def main(): - from invenio.legacy.bibformat.bibreformat import main as bibreformat_main - return bibreformat_main() diff --git a/invenio/legacy/bibformat/templates.py b/invenio/legacy/bibformat/templates.py deleted file mode 100644 index 9926adaa1..000000000 --- a/invenio/legacy/bibformat/templates.py +++ /dev/null @@ -1,53 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""HTML Templates for BibFormat administration""" - - -class Template(object): - """Templating class, refer to bibformat.py for examples of call""" - - def tmpl_admin_dialog_box(self, url, title, message, options): - """ - Prints a dialog box with given title, message and options - - @param url: the url of the page that must process the result of the dialog box - @param ln: language - @param title: the title of the dialog box - @param message: a formatted message to display inside dialog box - @param options: a list of string options to display as button to the user - @return: HTML markup - """ - - out = "" - out += ''' -
-
- %(title)s: -

%(message)s

-
- ''' % {'title': title, - 'message': message, - 'url': url} - - for option in options: - out += ''' ''' % {'value': option} - - out += '''
''' - return out diff --git a/invenio/legacy/webbasket/webinterface.py b/invenio/legacy/webbasket/webinterface.py index 20c8f86ab..89c1f77b0 100644 --- a/invenio/legacy/webbasket/webinterface.py +++ b/invenio/legacy/webbasket/webinterface.py @@ -1,1630 +1,1630 @@ # This file is part of Invenio. # Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """WebBasket Web Interface.""" __revision__ = "$Id$" __lastupdated__ = """$Date$""" from invenio.utils import apache import os import cgi import urllib from invenio.config import CFG_SITE_SECURE_URL, \ CFG_ACCESS_CONTROL_LEVEL_SITE, \ - CFG_SITE_SECURE_URL, CFG_PREFIX, CFG_SITE_LANG + CFG_SITE_SECURE_URL, CFG_DATADIR, CFG_SITE_LANG from invenio.base.globals import cfg from invenio.base.i18n import gettext_set_language from invenio.legacy.webpage import page from invenio.legacy.webuser import getUid, page_not_authorized, isGuestUser from invenio.config import CFG_BASKETSDIR from invenio.legacy.webbasket.api import \ check_user_can_comment, \ check_sufficient_rights, \ perform_request_display, \ perform_request_search, \ create_guest_warning_box, \ create_basket_navtrail, \ perform_request_write_note, \ perform_request_save_note, \ perform_request_delete_note, \ perform_request_add_group, \ perform_request_edit, \ perform_request_edit_topic, \ perform_request_list_public_baskets, \ perform_request_unsubscribe, \ perform_request_subscribe, \ perform_request_display_public, \ perform_request_write_public_note, \ perform_request_save_public_note, \ delete_record, \ move_record, \ perform_request_add, \ perform_request_create_basket, \ perform_request_delete, \ wash_topic, \ wash_group, \ perform_request_export_xml, \ page_start, \ page_end from invenio.legacy.webbasket.db_layer import get_basket_name, \ get_max_user_rights_on_basket from invenio.utils.url import get_referer, redirect_to_url, make_canonical_urlargd from invenio.ext.legacy.handler import wash_urlargd, WebInterfaceDirectory from invenio.legacy.webstat.api import register_customevent from invenio.ext.logging import register_exception from invenio.legacy.webuser import collect_user_info from invenio.modules.comments.api import check_user_can_attach_file_to_comments from invenio.modules.access.engine import acc_authorize_action from invenio.utils.html import is_html_text_editor_installed from invenio.legacy.ckeditor.connector import process_CKEditor_upload, send_response from invenio.legacy.bibdocfile.api import stream_file class WebInterfaceBasketCommentsFiles(WebInterfaceDirectory): """Handle upload and access to files for comments in WebBasket. The upload is currently only available through the CKEditor. """ def _lookup(self, component, path): """ This handler is invoked for the dynamic URLs (for getting and putting attachments) Eg: /yourbaskets/attachments/get/31/652/5/file/myfile.pdf /yourbaskets/attachments/get/31/552/5/image/myfigure.png bskid/recid/uid/ /yourbaskets/attachments/put/31/550/ bskid/recid """ if component == 'get' and len(path) > 4: bskid = path[0] # Basket id recid = path[1] # Record id uid = path[2] # uid of the submitter file_type = path[3] # file, image, flash or media (as # defined by CKEditor) if file_type in ['file', 'image', 'flash', 'media']: file_name = '/'.join(path[4:]) # the filename def answer_get(req, form): """Accessing files attached to comments.""" form['file'] = file_name form['type'] = file_type form['uid'] = uid form['recid'] = recid form['bskid'] = bskid return self._get(req, form) return answer_get, [] elif component == 'put' and len(path) > 1: bskid = path[0] # Basket id recid = path[1] # Record id def answer_put(req, form): """Attaching file to a comment.""" form['recid'] = recid form['bskid'] = bskid return self._put(req, form) return answer_put, [] # All other cases: file not found return None, [] def _get(self, req, form): """ Returns a file attached to a comment. A file is attached to a comment of a record of a basket, by a user (who is the author of the comment), and is of a certain type (file, image, etc). Therefore these 5 values are part of the URL. Eg: CFG_SITE_SECURE_URL/yourbaskets/attachments/get/31/91/5/file/myfile.pdf bskid/recid/uid """ argd = wash_urlargd(form, {'file': (str, None), 'type': (str, None), 'uid': (int, 0), 'bskid': (int, 0), 'recid': (int, 0)}) _ = gettext_set_language(argd['ln']) # Can user view this basket & record & comment, i.e. can user # access its attachments? #uid = getUid(req) user_info = collect_user_info(req) rights = get_max_user_rights_on_basket(argd['uid'], argd['bskid']) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if user_info['email'] == 'guest': # Ask to login target = CFG_SITE_SECURE_URL + '/youraccount/login' + \ make_canonical_urlargd({'ln' : argd['ln'], 'referer' : \ CFG_SITE_SECURE_URL + user_info['uri']}, {}) return redirect_to_url(req, target) elif not(check_sufficient_rights(rights, cfg['CFG_WEBBASKET_SHARE_LEVELS']['READITM'])): return page_not_authorized(req, "../", \ text = _("You are not authorized to view this attachment")) if not argd['file'] is None: # Prepare path to file on disk. Normalize the path so that # ../ and other dangerous components are removed. path = os.path.abspath(os.path.join(CFG_BASKETSDIR, 'comments', str(argd['bskid']), str(argd['recid']), str(argd['uid']), argd['type'], argd['file'])) # Check that we are really accessing attachements # directory, for the declared basket and record. if path.startswith(os.path.join(CFG_BASKETSDIR, 'comments', str(argd['bskid']), str(argd['recid']))) and \ os.path.exists(path): return stream_file(req, path) # Send error 404 in all other cases return apache.HTTP_NOT_FOUND def _put(self, req, form): """ Process requests received from CKEditor to upload files, etc. URL eg: CFG_SITE_SECURE_URL/yourbaskets/attachments/put/31/91/ bskid/recid/ """ if not is_html_text_editor_installed(): return argd = wash_urlargd(form, {'bskid': (int, 0), 'recid': (int, 0)}) uid = getUid(req) # URL where the file can be fetched after upload user_files_path = '%(CFG_SITE_SECURE_URL)s/yourbaskets/attachments/get/%(bskid)s/%(recid)i/%(uid)s' % \ {'uid': uid, 'recid': argd['recid'], 'bskid': argd['bskid'], 'CFG_SITE_SECURE_URL': CFG_SITE_SECURE_URL} # Path to directory where uploaded files are saved - user_files_absolute_path = '%(CFG_PREFIX)s/var/data/baskets/comments/%(bskid)s/%(recid)s/%(uid)s' % \ + user_files_absolute_path = '%(CFG_DATADIR)s/baskets/comments/%(bskid)s/%(recid)s/%(uid)s' % \ {'uid': uid, 'recid': argd['recid'], 'bskid': argd['bskid'], - 'CFG_PREFIX': CFG_PREFIX} + 'CFG_DATADIR': CFG_DATADIR} # Check that user can # 1. is logged in # 2. comment records of this basket (to simplify, we use # WebComment function to check this, even if it is not # entirely adequate) # 3. attach files user_info = collect_user_info(req) (auth_code, dummy) = check_user_can_attach_file_to_comments(user_info, argd['recid']) fileurl = '' callback_function = '' if user_info['email'] == 'guest': # 1. User is guest: must login prior to upload data ='Please login before uploading file.' if not user_info['precached_usebaskets']: msg = 'Sorry, you are not allowed to use WebBasket' elif not check_user_can_comment(uid, argd['bskid']): # 2. User cannot edit comment of this basket msg = 'Sorry, you are not allowed to submit files' elif auth_code: # 3. User cannot submit msg = 'Sorry, you are not allowed to submit files.' else: # Process the upload and get the response (msg, uploaded_file_path, filename, fileurl, callback_function) = \ process_CKEditor_upload(form, uid, user_files_path, user_files_absolute_path, recid=argd['recid']) send_response(req, msg, fileurl, callback_function) class WebInterfaceYourBasketsPages(WebInterfaceDirectory): """Defines the set of /yourbaskets pages.""" _exports = ['', 'display_item', 'display', 'search', 'write_note', 'save_note', 'delete_note', 'add', 'delete', 'modify', 'edit', 'edit_topic', 'create_basket', 'display_public', 'list_public_baskets', 'subscribe', 'unsubscribe', 'write_public_note', 'save_public_note', 'attachments'] attachments = WebInterfaceBasketCommentsFiles() def index(self, req, dummy): """Index page.""" redirect_to_url(req, '%s/yourbaskets/display?%s' % (CFG_SITE_SECURE_URL, req.args)) def display_item(self, req, dummy): """Legacy URL redirection.""" redirect_to_url(req, '%s/yourbaskets/display?%s' % (CFG_SITE_SECURE_URL, req.args)) def display(self, req, form): """Display basket interface.""" #import rpdb2; rpdb2.start_embedded_debugger('password', fAllowRemote=True) argd = wash_urlargd(form, {'category': (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic': (str, ""), 'group': (int, 0), 'bskid': (int, 0), 'recid': (int, 0), 'bsk_to_sort': (int, 0), 'sort_by_title': (str, ""), 'sort_by_date': (str, ""), 'of': (str, "hb"), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/display", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/display%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, dummy, navtrail) = perform_request_display(uid=uid, selected_category=argd['category'], selected_topic=argd['topic'], selected_group_id=argd['group'], selected_bskid=argd['bskid'], selected_recid=argd['recid'], of=argd['of'], ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body # register event in webstat if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["display", "", user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") rssurl = CFG_SITE_SECURE_URL + "/rss" if argd['of'] != 'hb': page_start(req, of=argd['of']) if argd['of'].startswith('x'): req.write(body) page_end(req, of=argd['of']) return elif argd['bskid']: rssurl = "%s/yourbaskets/display?category=%s&topic=%s&group=%i&bskid=%i&of=xr" % \ (CFG_SITE_SECURE_URL, argd['category'], urllib.quote(argd['topic']), argd['group'], argd['bskid']) return page(title = _("Display baskets"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1, rssurl=rssurl) def search(self, req, form): """Search baskets interface.""" argd = wash_urlargd(form, {'category': (str, ""), 'topic': (str, ""), 'group': (int, 0), 'p': (str, ""), 'b': (str, ""), 'n': (int, 0), 'of': (str, "hb"), 'verbose': (int, 0), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/search", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/search%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_search(uid=uid, selected_category=argd['category'], selected_topic=argd['topic'], selected_group_id=argd['group'], p=argd['p'], b=argd['b'], n=argd['n'], # format=argd['of'], ln=argd['ln']) # register event in webstat if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["search", "", user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Search baskets"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def write_note(self, req, form): """Write a comment (just interface for writing)""" argd = wash_urlargd(form, {'category': (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic': (str, ""), 'group': (int, 0), 'bskid': (int, 0), 'recid': (int, 0), 'cmtid': (int, 0), 'of' : (str, ''), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/write_note", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/write_note%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_write_note(uid=uid, category=argd['category'], topic=argd['topic'], group_id=argd['group'], bskid=argd['bskid'], recid=argd['recid'], cmtid=argd['cmtid'], ln=argd['ln']) # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["write_note", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Add a note"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def save_note(self, req, form): """Save comment on record in basket""" argd = wash_urlargd(form, {'category': (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic': (str, ""), 'group': (int, 0), 'bskid': (int, 0), 'recid': (int, 0), 'note_title': (str, ""), 'note_body': (str, ""), 'date_creation': (str, ""), 'editor_type': (str, ""), 'of': (str, ''), 'ln': (str, CFG_SITE_LANG), 'reply_to': (int, 0)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/save_note", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/save_note%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_save_note(uid=uid, category=argd['category'], topic=argd['topic'], group_id=argd['group'], bskid=argd['bskid'], recid=argd['recid'], note_title=argd['note_title'], note_body=argd['note_body'], date_creation=argd['date_creation'], editor_type=argd['editor_type'], ln=argd['ln'], reply_to=argd['reply_to']) # TODO: do not stat event if save was not succussful # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["save_note", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Display item and notes"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def delete_note(self, req, form): """Delete a comment @param bskid: id of basket (int) @param recid: id of record (int) @param cmtid: id of comment (int) @param category: category (see webbasket_config) (str) @param topic: nb of topic currently displayed (int) @param group: id of group baskets currently displayed (int) @param ln: language""" argd = wash_urlargd(form, {'category': (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic': (str, ""), 'group': (int, 0), 'bskid': (int, 0), 'recid': (int, 0), 'cmtid': (int, 0), 'of' : (str, ''), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/delete_note", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/delete_note%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/display%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_delete_note(uid=uid, category=argd['category'], topic=argd['topic'], group_id=argd['group'], bskid=argd['bskid'], recid=argd['recid'], cmtid=argd['cmtid'], ln=argd['ln']) # TODO: do not stat event if delete was not succussful # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) user_info = collect_user_info(req) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["delete_note", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Display item and notes"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def add(self, req, form): """Add records to baskets. @param recid: list of records to add @param colid: in case of external collections, the id of the collection the records belong to @param bskids: list of baskets to add records to. if not provided, will return a page where user can select baskets @param referer: URL of the referring page @param new_basket_name: add record to new basket @param new_topic_name: new basket goes into new topic @param create_in_topic: # of topic to put basket into @param ln: language""" # TODO: apply a maximum limit of items (100) that can be added to a basket # at once. Also see the build_search_url function of websearch_..._searcher.py # for the "rg" GET variable. argd = wash_urlargd(form, {'recid': (list, []), 'category': (str, ""), 'bskid': (int, 0), 'colid': (int, 0), 'es_title': (str, ""), 'es_desc': (str, ""), 'es_url': (str, ""), 'note_body': (str, ""), 'date_creation': (str, ""), 'editor_type': (str, ""), 'b': (str, ""), 'copy': (int, 0), 'move_from_basket': (int, 0), 'wait': (int, 0), 'referer': (str, ""), 'of': (str, ''), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/add", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/add%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if not argd['referer']: argd['referer'] = get_referer(req) (body, navtrail) = perform_request_add(uid=uid, recids=argd['recid'], colid=argd['colid'], bskid=argd['bskid'], es_title=argd['es_title'], es_desc=argd['es_desc'], es_url=argd['es_url'], note_body=argd['note_body'], date_creation=argd['date_creation'], editor_type=argd['editor_type'], category=argd['category'], b=argd['b'], copy=argd['copy'], move_from_basket=argd['move_from_basket'], wait=argd['wait'], referer=argd['referer'], ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body # register event in webstat bskid = argd['bskid'] basket_str = "%s (%s)" % (get_basket_name(bskid), bskid) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["add", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _('Add to basket'), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def delete(self, req, form): """Delete basket interface""" argd = wash_urlargd(form, {'bskid' : (int, -1), 'confirmed' : (int, 0), 'category' : (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic' : (str, ""), 'group' : (int, 0), 'of' : (str, ''), 'ln' : (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/delete", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/delete%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) body=perform_request_delete(uid=uid, bskid=argd['bskid'], confirmed=argd['confirmed'], category=argd['category'], selected_topic=argd['topic'], selected_group_id=argd['group'], ln=argd['ln']) if argd['confirmed']: if argd['category'] == cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']: argd['topic'] = wash_topic(uid, argd['topic'])[0] elif argd['category'] == cfg['CFG_WEBBASKET_CATEGORIES']['GROUP']: argd['group'] = wash_group(uid, argd['group'])[0] url = """%s/yourbaskets/display?category=%s&topic=%s&group=%i&ln=%s""" % \ (CFG_SITE_SECURE_URL, argd['category'], urllib.quote(argd['topic']), argd['group'], argd['ln']) redirect_to_url(req, url) else: navtrail = ''\ '%s' navtrail %= (CFG_SITE_SECURE_URL, argd['ln'], _("Your Account")) navtrail_end = create_basket_navtrail(uid=uid, category=argd['category'], topic=argd['topic'], group=argd['group'], bskid=argd['bskid'], ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["delete", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Delete a basket"), body = body, navtrail = navtrail + navtrail_end, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def modify(self, req, form): """Modify basket content interface (reorder, suppress record, etc.)""" argd = wash_urlargd(form, {'action': (str, ""), 'bskid': (int, -1), 'recid': (int, 0), 'category': (str, cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE']), 'topic': (str, ""), 'group': (int, 0), 'of' : (str, ''), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/modify", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/modify%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) url = CFG_SITE_SECURE_URL url += '/yourbaskets/display?category=%s&topic=%s&group=%i&bskid=%i&ln=%s' % \ (argd['category'], urllib.quote(argd['topic']), argd['group'], argd['bskid'], argd['ln']) if argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['DELETE']: delete_record(uid, argd['bskid'], argd['recid']) redirect_to_url(req, url) elif argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['UP']: move_record(uid, argd['bskid'], argd['recid'], argd['action']) redirect_to_url(req, url) elif argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['DOWN']: move_record(uid, argd['bskid'], argd['recid'], argd['action']) redirect_to_url(req, url) elif argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['COPY'] or \ argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['MOVE']: if(argd['action'] == cfg['CFG_WEBBASKET_ACTIONS']['MOVE']): title = _("Move record to basket") from_bsk = argd['bskid'] else: title = _("Copy record to basket") from_bsk = 0 referer = get_referer(req) (body, navtrail) = perform_request_add(uid=uid, recids=argd['recid'], copy=True, move_from_basket=from_bsk, referer=referer, ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body else: title = '' body = '' # warnings = [('WRN_WEBBASKET_UNDEFINED_ACTION',)] navtrail = ''\ '%s' navtrail %= (CFG_SITE_SECURE_URL, argd['ln'], _("Your Account")) navtrail_end = create_basket_navtrail(uid=uid, category=argd['category'], topic=argd['topic'], group=argd['group'], bskid=argd['bskid'], ln=argd['ln']) # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["modify", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = title, body = body, navtrail = navtrail + navtrail_end, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def edit(self, req, form): """Edit basket interface""" argd = wash_urlargd(form, {'bskid': (int, 0), 'groups': (list, []), 'topic': (str, ""), 'add_group': (str, ""), 'group_cancel': (str, ""), 'submit': (str, ""), 'cancel': (str, ""), 'delete': (str, ""), 'new_name': (str, ""), 'new_topic': (str, ""), 'new_topic_name': (str, ""), 'new_group': (str, ""), 'external': (str, ""), 'of' : (str, ''), 'ln': (str, CFG_SITE_LANG)}) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/edit", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/edit%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) _ = gettext_set_language(argd['ln']) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if argd['cancel']: url = CFG_SITE_SECURE_URL + '/yourbaskets/display?category=%s&topic=%s&ln=%s' url %= (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], urllib.quote(argd['topic']), argd['ln']) redirect_to_url(req, url) elif argd['delete']: url = CFG_SITE_SECURE_URL url += '/yourbaskets/delete?bskid=%i&category=%s&topic=%s&ln=%s' % \ (argd['bskid'], cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], urllib.quote(argd['topic']), argd['ln']) redirect_to_url(req, url) elif argd['add_group'] and not(argd['new_group']): body = perform_request_add_group(uid=uid, bskid=argd['bskid'], topic=argd['topic'], ln=argd['ln']) # warnings = [] elif (argd['add_group'] and argd['new_group']) or argd['group_cancel']: if argd['add_group']: perform_request_add_group(uid=uid, bskid=argd['bskid'], topic=argd['topic'], group_id=argd['new_group'], ln=argd['ln']) body = perform_request_edit(uid=uid, bskid=argd['bskid'], topic=argd['topic'], ln=argd['ln']) elif argd['submit']: body = perform_request_edit(uid=uid, bskid=argd['bskid'], topic=argd['topic'], new_name=argd['new_name'], new_topic=argd['new_topic'], new_topic_name=argd['new_topic_name'], groups=argd['groups'], external=argd['external'], ln=argd['ln']) if argd['new_topic'] != "-1": argd['topic'] = argd['new_topic'] url = CFG_SITE_SECURE_URL + '/yourbaskets/display?category=%s&topic=%s&ln=%s' % \ (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], urllib.quote(argd['topic']), argd['ln']) redirect_to_url(req, url) else: body = perform_request_edit(uid=uid, bskid=argd['bskid'], topic=argd['topic'], ln=argd['ln']) navtrail = ''\ '%s' navtrail %= (CFG_SITE_SECURE_URL, argd['ln'], _("Your Account")) navtrail_end = create_basket_navtrail( uid=uid, category=cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], topic=argd['topic'], group=0, bskid=argd['bskid'], ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["edit", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Edit basket"), body = body, navtrail = navtrail + navtrail_end, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def edit_topic(self, req, form): """Edit topic interface""" argd = wash_urlargd(form, {'topic': (str, ""), 'submit': (str, ""), 'cancel': (str, ""), 'delete': (str, ""), 'new_name': (str, ""), 'of' : (str, ''), 'ln': (str, CFG_SITE_LANG)}) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/edit", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/edit_topic%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) _ = gettext_set_language(argd['ln']) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if argd['cancel']: url = CFG_SITE_SECURE_URL + '/yourbaskets/display?category=%s&ln=%s' url %= (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], argd['ln']) redirect_to_url(req, url) elif argd['delete']: url = CFG_SITE_SECURE_URL url += '/yourbaskets/delete?bskid=%i&category=%s&topic=%s&ln=%s' % \ (argd['bskid'], cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], urllib.quote(argd['topic']), argd['ln']) redirect_to_url(req, url) elif argd['submit']: body = perform_request_edit_topic(uid=uid, topic=argd['topic'], new_name=argd['new_name'], ln=argd['ln']) url = CFG_SITE_SECURE_URL + '/yourbaskets/display?category=%s&ln=%s' % \ (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], argd['ln']) redirect_to_url(req, url) else: body = perform_request_edit_topic(uid=uid, topic=argd['topic'], ln=argd['ln']) navtrail = ''\ '%s' navtrail %= (CFG_SITE_SECURE_URL, argd['ln'], _("Your Account")) navtrail_end = "" #navtrail_end = create_basket_navtrail( # uid=uid, # category=cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], # topic=argd['topic'], # group=0, # ln=argd['ln']) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body # register event in webstat #basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) #if user_info['email']: # user_str = "%s (%d)" % (user_info['email'], user_info['uid']) #else: # user_str = "" #try: # register_customevent("baskets", ["edit", basket_str, user_str]) #except: # register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Edit topic"), body = body, navtrail = navtrail + navtrail_end, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def create_basket(self, req, form): """Create basket interface""" argd = wash_urlargd(form, {'new_basket_name': (str, ""), 'new_topic_name' : (str, ""), 'create_in_topic': (str, "-1"), 'topic' : (str, ""), 'recid' : (list, []), 'colid' : (int, -1), 'es_title' : (str, ''), 'es_desc' : (str, ''), 'es_url' : (str, ''), 'copy' : (int, 0), 'move_from_basket':(int, 0), 'referer' : (str, ''), 'of' : (str, ''), 'ln' : (str, CFG_SITE_LANG)}) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/create_basket", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/create_basket%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) _ = gettext_set_language(argd['ln']) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if argd['new_basket_name'] and \ (argd['new_topic_name'] or argd['create_in_topic'] != "-1"): (bskid, topic) = perform_request_create_basket( req, uid=uid, new_basket_name=argd['new_basket_name'], new_topic_name=argd['new_topic_name'], create_in_topic=argd['create_in_topic'], recids=argd['recid'], colid=argd['colid'], es_title=argd['es_title'], es_desc=argd['es_desc'], es_url=argd['es_url'], copy=argd['copy'], move_from_basket=argd['move_from_basket'], referer=argd['referer'], ln=argd['ln']) # register event in webstat basket_str = "%s ()" % argd['new_basket_name'] if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["create_basket", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") if ( argd['recid'] and argd['colid'] >= 0 ): url = CFG_SITE_SECURE_URL + '/yourbaskets/add?category=%s©=%i&referer=%s&bskid=%i&colid=%i&move_from_basket=%i&recid=%s&wait=1&ln=%s' url %= (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], argd['copy'], urllib.quote(argd['referer']), bskid, argd['colid'], argd['move_from_basket'], '&recid='.join(str(recid) for recid in argd['recid']), argd['ln']) elif ( argd['es_title'] and argd['es_desc'] and argd['es_url'] and argd['colid'] == -1 ): # Adding NEW external record - this does not need 'move_from_basket' data url = CFG_SITE_SECURE_URL + '/yourbaskets/add?category=%s&bskid=%i&colid=%i&es_title=%s&es_desc=%s&es_url=%s&wait=1&ln=%s' url %= (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], bskid, argd['colid'], urllib.quote(argd['es_title']), urllib.quote(argd['es_desc']), urllib.quote(argd['es_url']), argd['ln']) else: url = CFG_SITE_SECURE_URL + '/yourbaskets/display?category=%s&topic=%s&ln=%s' url %= (cfg['CFG_WEBBASKET_CATEGORIES']['PRIVATE'], urllib.quote(topic), argd['ln']) redirect_to_url(req, url) else: body = perform_request_create_basket(req, uid=uid, new_basket_name=argd['new_basket_name'], new_topic_name=argd['new_topic_name'], create_in_topic=argd['create_in_topic'], topic=argd['topic'], recids=argd['recid'], colid=argd['colid'], es_title=argd['es_title'], es_desc=argd['es_desc'], es_url=argd['es_url'], copy=argd['copy'], move_from_basket=argd['move_from_basket'], referer=argd['referer'], ln=argd['ln']) navtrail = '%s' navtrail %= (CFG_SITE_SECURE_URL, argd['ln'], _("Your Account")) if isGuestUser(uid): body = create_guest_warning_box(argd['ln']) + body return page(title = _("Create basket"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def display_public(self, req, form): """Display a public basket""" argd = wash_urlargd(form, {'bskid': (int, 0), 'recid': (int, 0), 'of': (str, "hb"), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/display", navmenuid = 'yourbaskets') user_info = collect_user_info(req) if not argd['bskid']: (body, navtrail) = perform_request_list_public_baskets(uid) title = _('List of public baskets') # register event in webstat if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["list_public_baskets", "", user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") else: (body, dummy, navtrail) = perform_request_display_public(uid=uid, selected_bskid=argd['bskid'], selected_recid=argd['recid'], of=argd['of'], ln=argd['ln']) title = _('Public basket') # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["display_public", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") rssurl = CFG_SITE_SECURE_URL + "/rss" if argd['of'] != 'hb': page_start(req, of=argd['of']) if argd['of'].startswith('x'): req.write(body) page_end(req, of=argd['of']) return elif argd['bskid']: rssurl = "%s/yourbaskets/display_public?&bskid=%i&of=xr" % \ (CFG_SITE_SECURE_URL, argd['bskid']) return page(title = title, body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1, rssurl=rssurl) def list_public_baskets(self, req, form): """List of public baskets interface.""" argd = wash_urlargd(form, {'limit': (int, 1), 'sort': (str, 'name'), 'asc': (int, 1), 'of': (str, ''), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE == 2: return page_not_authorized(req, "../yourbaskets/list_public_baskets", navmenuid = 'yourbaskets') user_info = collect_user_info(req) nb_views_show = acc_authorize_action(user_info, 'runwebstatadmin') nb_views_show_p = not(nb_views_show[0]) (body, navtrail) = perform_request_list_public_baskets(uid, argd['limit'], argd['sort'], argd['asc'], nb_views_show_p, argd['ln']) return page(title = _("List of public baskets"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def subscribe(self, req, form): """Subscribe to a basket pseudo-interface.""" argd = wash_urlargd(form, {'bskid': (int, 0), 'of': (str, 'hb'), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE == 2: return page_not_authorized(req, "../yourbaskets/subscribe", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/subscribe%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if not argd['bskid']: (body, navtrail) = perform_request_list_public_baskets(uid) title = _('List of public baskets') else: # TODO: Take care of XML output as shown below #req.content_type = "text/xml" #req.send_http_header() #return perform_request_display_public(bskid=argd['bskid'], of=argd['of'], ln=argd['ln']) subscribe_warnings_html = perform_request_subscribe(uid, argd['bskid'], argd['ln']) (body, dummy, navtrail) = perform_request_display_public(uid=uid, selected_bskid=argd['bskid'], selected_recid=0, of=argd['of'], ln=argd['ln']) #warnings.extend(subscribe_warnings) body = subscribe_warnings_html + body title = _('Public basket') return page(title = title, body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def unsubscribe(self, req, form): """Unsubscribe from basket pseudo-interface.""" argd = wash_urlargd(form, {'bskid': (int, 0), 'of': (str, 'hb'), 'ln': (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE == 2: return page_not_authorized(req, "../yourbaskets/unsubscribe", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/unsubscribe%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) if not argd['bskid']: (body, navtrail) = perform_request_list_public_baskets(uid) title = _('List of public baskets') else: # TODO: Take care of XML output as shown below #req.content_type = "text/xml" #req.send_http_header() #return perform_request_display_public(bskid=argd['bskid'], of=argd['of'], ln=argd['ln']) unsubscribe_warnings_html = perform_request_unsubscribe(uid, argd['bskid'], argd['ln']) (body, dummy, navtrail) = perform_request_display_public(uid=uid, selected_bskid=argd['bskid'], selected_recid=0, of=argd['of'], ln=argd['ln']) # warnings.extend(unsubscribe_warnings) body = unsubscribe_warnings_html + body title = _('Public basket') return page(title = title, body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) def write_public_note(self, req, form): """Write a comment (just interface for writing)""" argd = wash_urlargd(form, {'bskid': (int, 0), 'recid': (int, 0), 'cmtid': (int, 0), 'of' : (str, ''), 'ln' : (str, CFG_SITE_LANG)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/write_public_note", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/write_public_note%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_write_public_note(uid=uid, bskid=argd['bskid'], recid=argd['recid'], cmtid=argd['cmtid'], ln=argd['ln']) # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["write_public_note", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Add a note"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], secure_page_p=1) def save_public_note(self, req, form): """Save comment on record in basket""" argd = wash_urlargd(form, {'bskid': (int, 0), 'recid': (int, 0), 'note_title': (str, ""), 'note_body': (str, ""), 'editor_type': (str, ""), 'of': (str, ''), 'ln': (str, CFG_SITE_LANG), 'reply_to': (str, 0)}) _ = gettext_set_language(argd['ln']) uid = getUid(req) if uid == -1 or CFG_ACCESS_CONTROL_LEVEL_SITE >= 1: return page_not_authorized(req, "../yourbaskets/save_public_note", navmenuid = 'yourbaskets') if isGuestUser(uid): return redirect_to_url(req, "%s/youraccount/login%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd({ 'referer' : "%s/yourbaskets/save_public_note%s" % ( CFG_SITE_SECURE_URL, make_canonical_urlargd(argd, {})), "ln" : argd['ln']}, {}))) user_info = collect_user_info(req) if not user_info['precached_usebaskets']: return page_not_authorized(req, "../", \ text = _("You are not authorized to use baskets.")) (body, navtrail) = perform_request_save_public_note(uid=uid, bskid=argd['bskid'], recid=argd['recid'], note_title=argd['note_title'], note_body=argd['note_body'], editor_type=argd['editor_type'], ln=argd['ln'], reply_to=argd['reply_to']) # TODO: do not stat event if save was not succussful # register event in webstat basket_str = "%s (%d)" % (get_basket_name(argd['bskid']), argd['bskid']) if user_info['email']: user_str = "%s (%d)" % (user_info['email'], user_info['uid']) else: user_str = "" try: register_customevent("baskets", ["save_public_note", basket_str, user_str]) except: register_exception(suffix="Do the webstat tables exists? Try with 'webstatadmin --load-config'") return page(title = _("Display item and notes"), body = body, navtrail = navtrail, uid = uid, lastupdated = __lastupdated__, language = argd['ln'], req = req, navmenuid = 'yourbaskets', of = argd['of'], navtrail_append_title_p = 0, secure_page_p=1) diff --git a/invenio/modules/formatter/__init__.py b/invenio/modules/formatter/__init__.py index 7c1dc827a..1ea24f320 100644 --- a/invenio/modules/formatter/__init__.py +++ b/invenio/modules/formatter/__init__.py @@ -1,175 +1,164 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Format records using chosen format.""" from invenio.base.globals import cfg def format_record(recID, of, ln=None, verbose=0, search_pattern=None, xml_record=None, user_info=None, on_the_fly=False, save_missing=True, force_2nd_pass=False, **kwargs): """Format a record in given output format. Return a formatted version of the record in the specified language, search pattern, and with the specified output format. The function will define which format template must be applied. The record to be formatted can be specified with its ID (with 'recID' parameter) or given as XML representation (with 'xml_record' parameter). If 'xml_record' is specified 'recID' is ignored (but should still be given for reference. A dummy recid 0 or -1 could be used). 'user_info' allows to grant access to some functionalities on a page depending on the user's priviledges. The 'user_info' object makes sense only in the case of on-the-fly formatting. 'user_info' is the same object as the one returned by 'webuser.collect_user_info(req)' :param recID: the ID of record to format. :type recID: int :param of: an output format code (or short identifier for the output format) :type of: string :param ln: the language to use to format the record :type ln: string :param verbose: the level of verbosity from 0 to 9. - O: silent - 5: errors - 7: errors and warnings, stop if error in format elements - 9: errors and warnings, stop if error (debug mode) :type verbose: int :param search_pattern: list of strings representing the user request in web interface :type search_pattern: list(string) :param xml_record: an xml string represention of the record to format :type xml_record: string or None :param user_info: the information of the user who will view the formatted page (if applicable) :param on_the_fly: if False, try to return an already preformatted version of the record in the database :type on_the_fly: boolean :return: formatted record :rtype: string """ ln = ln or cfg['CFG_SITE_LANG'] from . import engine as bibformat_engine out, needs_2nd_pass = bibformat_engine.format_record_1st_pass( recID=recID, of=of, ln=ln, verbose=verbose, search_pattern=search_pattern, xml_record=xml_record, user_info=user_info, on_the_fly=on_the_fly, save_missing=save_missing, **kwargs) - if needs_2nd_pass or force_2nd_pass: - out = bibformat_engine.format_record_2nd_pass( - recID=recID, - of=of, - template=out, - ln=ln, - verbose=verbose, - search_pattern=search_pattern, - xml_record=xml_record, - user_info=user_info, - **kwargs) return out def format_with_format_template(format_template_filename, bfo, verbose=0, format_template_code=None): """Wrapper around format template.""" from . import engine as bibformat_engine evaluated_format, dummy = bibformat_engine.format_with_format_template( format_template_filename=format_template_filename, bfo=bfo, verbose=verbose, format_template_code=format_template_code) return evaluated_format def get_output_format_content_type(of, default_content_type="text/html"): """ Return the content type of the given output format. For example `text/html` or `application/ms-excel`. :param of: the code of output format for which we want to get the content type :param default_content_type: default content-type when content-type was not set up :return: the content-type to use for this output format """ from . import api content_type = api.get_output_format_content_type(of) if content_type == '': content_type = default_content_type return content_type def print_records(recIDs, of='hb', ln=None, verbose=0, search_pattern='', on_the_fly=False, **ctx): """Return records using Jinja template.""" import time from math import ceil from flask import request from invenio.base.i18n import wash_language from invenio.ext.template import render_template_to_string from invenio.utils.pagination import Pagination from .registry import export_formats from .engine import TEMPLATE_CONTEXT_FUNCTIONS_CACHE of = of.lower() jrec = request.values.get('jrec', ctx.get('jrec', 1), type=int) rg = request.values.get('rg', ctx.get('rg', 10), type=int) ln = ln or wash_language(request.values.get('ln', cfg['CFG_SITE_LANG'])) ot = (request.values.get('ot', ctx.get('ot')) or '').split(',') records = ctx.get('records', len(recIDs)) if jrec > records: jrec = rg * (records // rg) + 1 pages = int(ceil(jrec / float(rg))) if rg > 0 else 1 context = dict( of=of, jrec=jrec, rg=rg, ln=ln, ot=ot, facets={}, time=time, recids=recIDs, pagination=Pagination(pages, rg, records), verbose=verbose, export_formats=export_formats, format_record=format_record, **TEMPLATE_CONTEXT_FUNCTIONS_CACHE.template_context_functions ) context.update(ctx) return render_template_to_string( ['format/records/%s.tpl' % of, 'format/records/%s.tpl' % of[0], 'format/records/%s.tpl' % get_output_format_content_type(of). replace('/', '_')], **context) diff --git a/invenio/modules/formatter/engine.py b/invenio/modules/formatter/engine.py index d06113266..6f76273a4 100644 --- a/invenio/modules/formatter/engine.py +++ b/invenio/modules/formatter/engine.py @@ -1,2096 +1,1064 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Format a single record using specified format. You can have a look at the various escaping modes available in X{BibFormatObject} in function L{escape_field} Still it is useful sometimes for debugging purpose to use the L{BibFormatObject} class directly. For eg: >>> from invenio.modules.formatter.engine import BibFormatObject >>> bfo = BibFormatObject(102) >>> bfo.field('245__a') The order Rodentia in South America >>> from invenio.modules.formatter.format_elements import bfe_title >>> bfe_title.format_element(bfo) The order Rodentia in South America @see: bibformat.py, bibformat_utils.py """ import cgi -import inspect import os import re -import sys -import traceback import types -import warnings + from HTMLParser import HTMLParseError -from operator import itemgetter -from flask import current_app, has_app_context -from six import iteritems +from flask import current_app from werkzeug.utils import cached_property -import invenio.legacy.bibformat.dblayer as bibformat_dblayer from invenio.base.globals import cfg from invenio.base.i18n import gettext_set_language, language_list_long, \ wash_language from invenio.config import CFG_BIBFORMAT_CACHED_FORMATS, \ CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS, CFG_BIBFORMAT_HIDDEN_TAGS, \ CFG_SITE_LANG from invenio.ext.logging import register_exception from invenio.ext.template import render_template_to_string from invenio.legacy.bibrecord import create_record, record_empty, \ record_get_field_instances, record_get_field_value, \ record_get_field_values, record_xml_output from invenio.modules.access.engine import acc_authorize_action from invenio.modules.formatter.registry import template_context_functions from invenio.modules.formatter.utils import parse_tag from invenio.modules.knowledge.api import get_kbr_values -from invenio.utils.deprecation import RemovedInInvenio22Warning from invenio.utils.html import CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST, \ CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST, HTMLWasher -from invenio.utils.memoise import memoize from . import registry from .config import CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION, \ CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION, \ CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, CFG_BIBFORMAT_OUTPUTS_PATH, \ CFG_BIBFORMAT_TEMPLATES_DIR, InvenioBibFormatError from .engines import xslt # Cache for data we have already read and parsed format_templates_cache = {} -format_elements_cache = {} format_outputs_cache = {} html_field = '' # String indicating that field should be # treated as HTML (and therefore no escaping of # HTML tags should occur. # Appears in some field values. washer = HTMLWasher() # Used to remove dangerous tags from HTML # sources # Regular expression for finding ... tag in format templates pattern_lang = re.compile(r''' #closing start tag (?P.*?) #anything but the next group (greedy) () #end tag ''', re.IGNORECASE | re.DOTALL | re.VERBOSE) # Builds regular expression for finding each known language in tags ln_pattern_text = r"<(" for lang in language_list_long(enabled_langs_only=False): ln_pattern_text += lang[0] +r"|" ln_pattern_text = ln_pattern_text.rstrip(r"|") ln_pattern_text += r")>(.*?)" ln_pattern = re.compile(ln_pattern_text, re.IGNORECASE | re.DOTALL) # Regular expression for finding text to be translated TRANSLATION_PATTERN = re.compile(r'_\((?P.*?)\)_', re.IGNORECASE | re.DOTALL | re.VERBOSE) # Regular expression for finding tag in format templates pattern_format_template_name = re.compile(r''' #closing start tag (?P.*?) #name value. any char that is not end tag ()(\n)? #end tag ''', re.IGNORECASE | re.DOTALL | re.VERBOSE) # Regular expression for finding tag in format templates pattern_format_template_desc = re.compile(r''' #closing start tag (?P.*?) #description value. any char that is not end tag (\n)? #end tag ''', re.IGNORECASE | re.DOTALL | re.VERBOSE) # Regular expression for finding tags in format templates pattern_tag = re.compile(r''' [^/\s]+) #any char but a space or slash \s* #any number of spaces (?P(\s* #params here (?P([^=\s])*)\s* #param name: any chars that is not a white space or equality. Followed by space(s) =\s* #equality: = followed by any number of spaces (?P[\'"]) #one of the separators (?P.*?) #param value: any chars that is not a separator like previous one (?P=sep) #same separator as starting one )*) #many params \s* #any number of spaces (/)?> #end of the tag ''', re.IGNORECASE | re.DOTALL | re.VERBOSE) # Regular expression for finding params inside tags in format templates pattern_function_params = re.compile(r''' (?P([^=\s])*)\s* # Param name: any chars that is not a white space or equality. Followed by space(s) =\s* # Equality: = followed by any number of spaces (?P[\'"]) # One of the separators (?P.*?) # Param value: any chars that is not a separator like previous one (?P=sep) # Same separator as starting one ''', re.VERBOSE | re.DOTALL) # Regular expression for finding format elements "params" attributes # (defined by @param) pattern_format_element_params = re.compile(r''' @param\s* # Begins with AT param keyword followed by space(s) (?P[^\s=]*):\s* # A single keyword and comma, then space(s) #(=\s*(?P[\'"]) # Equality, space(s) and then one of the separators #(?P.*?) # Default value: any chars that is not a separator like previous one #(?P=sep) # Same separator as starting one #)?\s* # Default value for param is optional. Followed by space(s) (?P.*) # Any text that is not end of line (thanks to MULTILINE parameter) ''', re.VERBOSE | re.MULTILINE) # Regular expression for finding format elements "see also" attribute # (defined by @see) pattern_format_element_seealso = re.compile(r'''@see:\s*(?P.*)''', re.VERBOSE | re.MULTILINE) #Regular expression for finding 2 expressions in quotes, separated by #comma (as in template("1st","2nd") ) #Used when parsing output formats # pattern_parse_tuple_in_quotes = re.compile(''' # (?P[\'"]) # (?P.*) # (?P=sep1) # \s*,\s* # (?P[\'"]) # (?P.*) # (?P=sep2) # ''', re.VERBOSE | re.MULTILINE) sub_non_alnum = re.compile('[^0-9a-zA-Z]+') fix_tag_name = lambda s: sub_non_alnum.sub('_', s.lower()) - - class LazyTemplateContextFunctionsCache(object): """Loads bibformat elements using plugin builder and caches results.""" @cached_property def template_context_functions(self): """Returns template context functions""" modules = template_context_functions elem = {} for m in modules: register_func = getattr(m, 'template_context_function', None) if register_func and isinstance(register_func, types.FunctionType): elem[m.__name__.split('.')[-1]] = register_func return elem - @memoize - def bibformat_elements(self, modules=None): - """Returns bibformat elements.""" - if modules is None: - modules = registry.format_elements - - elem = {} - for m in modules: - if m is None: - continue - name = m.__name__.split('.')[-1] - if name in elem: - continue - filename = m.__file__[:-1] if m.__file__.endswith('.pyc') \ - else m.__file__ - register_func = getattr(m, 'format_element', - getattr(m, 'format', None)) - escape_values = getattr(m, 'escape_values', None) - if register_func and isinstance(register_func, types.FunctionType): - register_func._escape_values = escape_values - register_func.__file__ = filename - elem[name] = register_func - - return elem - - @cached_property - def functions(self): - - def insert(name): - def _bfe_element(bfo, **kwargs): - warnings.warn( - "BFE elements ({0}) are deprecated. " - "Use Jinja2 filters or macros.".format(name), - RemovedInInvenio22Warning - ) - # convert to utf-8 for legacy app - kwargs = dict((k, v.encode('utf-8') if isinstance(v, unicode) else v) - for k, v in iteritems(kwargs)) - format_element = get_format_element(name) - (out, dummy) = eval_format_element(format_element, - bfo, - kwargs) - # returns unicode for jinja2 - return out.decode('utf-8') - return _bfe_element - - # Old bibformat templates - tfn_from_files = dict((name.lower(), insert(name.lower())) - for name in self.bibformat_elements().keys()) - # Update with new template context functions - tfn_from_files.update(self.template_context_functions) - - bfe_from_tags = {} - if has_app_context(): - from invenio.ext.sqlalchemy import db - from invenio.modules.search.models import Tag - - # get functions from tag table - bfe_from_tags = dict(('bfe_'+fix_tag_name(name), - insert(fix_tag_name(name))) - for name in map(itemgetter(0), - db.session.query(Tag.name).all())) - - # overwrite functions from tag table with functions from files - bfe_from_tags.update(tfn_from_files) - return bfe_from_tags - TEMPLATE_CONTEXT_FUNCTIONS_CACHE = LazyTemplateContextFunctionsCache() -def get_format_element_path(filename): - if filename.endswith('.py'): - filename = filename[:-3] - return TEMPLATE_CONTEXT_FUNCTIONS_CACHE.bibformat_elements()[filename].__file__ - - def format_record(recID, of, ln=CFG_SITE_LANG, verbose=0, search_pattern=None, xml_record=None, user_info=None, qid="", **kwargs): """ Formats a record given output format. Main entry function of bibformat engine. Returns a formatted version of the record in the specified language, search pattern, and with the specified output format. The function will define which format template must be applied. You can either specify an record ID to format, or give its xml representation. if 'xml_record' is not None, then use it instead of recID. 'user_info' allows to grant access to some functionalities on a page depending on the user's priviledges. 'user_info' is the same object as the one returned by 'webuser.collect_user_info(req)' :param recID: the ID of record to format :param of: an output format code (or short identifier for the output format) :param ln: the language to use to format the record :param verbose: the level of verbosity from 0 to 9 (O: silent, 5: errors, 7: errors and warnings, stop if error in format elements 9: errors and warnings, stop if error (debug mode )) :param search_pattern: list of strings representing the user request in web interface :param xml_record: an xml string representing the record to format :param user_info: the information of the user who will view the formatted page @return: formatted record """ if search_pattern is None: search_pattern = [] out = "" ln = wash_language(ln) _ = gettext_set_language(ln) # Temporary workflow (during migration of formats): # Call new BibFormat # But if format not found for new BibFormat, then call old BibFormat #Create a BibFormat Object to pass that contain record and context bfo = BibFormatObject(recID, ln, search_pattern, xml_record, user_info, of) if of.lower() != 'xm' and (not bfo.get_record() or record_empty(bfo.get_record())): # Record only has recid: do not format, excepted # for xm format return "", False #Find out which format template to use based on record and output format. template = decide_format_template(bfo, of) - if verbose == 9 and template is not None: - out += """\n
- Using %s template for record %i. - """ % (template, recID) path = registry.format_templates_lookup.get(template) if template is None or not ( template.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION) or path is None or os.access(path, os.R_OK) ): - # template not found in new BibFormat. Call old one - if verbose == 9: - if template is None: - out += """\n
- No template found for output format %s and record %i. - (Check invenio.err log file for more details) - """ % (of, recID) - else: - out += """\n
- Template %s could not be read. - """ % (template) try: raise InvenioBibFormatError(_('No template could be found for output format %(code)s.', code=of)) except InvenioBibFormatError as exc: register_exception(req=bfo.req) - if verbose > 5: - out += """\n
- %s - """ % str(exc) - return out, False # Format with template out_, needs_2nd_pass = format_with_format_template( template, bfo, verbose=verbose, extra_context=kwargs) out += out_ return out, needs_2nd_pass def format_record_1st_pass(recID, of, ln=CFG_SITE_LANG, verbose=0, search_pattern=None, xml_record=None, user_info=None, on_the_fly=False, save_missing=True, **kwargs): """ Format a record in given output format. Return a formatted version of the record in the specified language, search pattern, and with the specified output format. The function will define which format template must be applied. The record to be formatted can be specified with its ID (with 'recID' parameter) or given as XML representation (with 'xml_record' parameter). If 'xml_record' is specified 'recID' is ignored (but should still be given for reference. A dummy recid 0 or -1 could be used). 'user_info' allows to grant access to some functionalities on a page depending on the user's priviledges. The 'user_info' object makes sense only in the case of on-the-fly formatting. 'user_info' is the same object as the one returned by 'webuser.collect_user_info(req)' :param recID: the ID of record to format. @type recID: int :param of: an output format code (or short identifier for the output format) @type of: string :param ln: the language to use to format the record @type ln: string :param verbose: the level of verbosity from 0 to 9 (O: silent, 5: errors, 7: errors and warnings, stop if error in format elements 9: errors and warnings, stop if error (debug mode )) @type verbose: int :param search_pattern: list of strings representing the user request in web interface @type search_pattern: list(string) :param xml_record: an xml string represention of the record to format @type xml_record: string or None :param user_info: the information of the user who will view the formatted page (if applicable) :param on_the_fly: if False, try to return an already preformatted version of the record in the database @type on_the_fly: boolean @return: formatted record @rtype: string """ - from invenio.legacy.search_engine import record_exists if search_pattern is None: search_pattern = [] out = "" - if verbose == 9: - out += """\n - Formatting record %i with output format %s. - """ % (recID, of) - if not on_the_fly and \ - (ln == CFG_SITE_LANG or - of.lower() == 'xm' or - (of.lower() in CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS)) and \ - record_exists(recID) != -1: - # Try to fetch preformatted record. Only possible for records - # formatted in CFG_SITE_LANG language (other are never - # stored), or of='xm' which does not depend on language. - # Exceptions are made for output formats defined in - # CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS, which are - # always served from the same cache for any language. Also, - # do not fetch from DB when record has been deleted: we want - # to return an "empty" record in that case - res, needs_2nd_pass = bibformat_dblayer.get_preformatted_record(recID, of) - if res is not None: - # record 'recID' is formatted in 'of', so return it - if verbose == 9: - last_updated = bibformat_dblayer.get_preformatted_record_date(recID, of) - out += """\n
- Found preformatted output for record %i (cache updated on %s). -
""" % (recID, last_updated) - if of.lower() == 'xm': - res = filter_hidden_fields(res, user_info) - # try to replace language links in pre-cached res, if applicable: - if ln != CFG_SITE_LANG and of.lower() in CFG_BIBFORMAT_DISABLE_I18N_FOR_CACHED_FORMATS: - # The following statements try to quickly replace any - # language arguments in URL links. Not an exact - # science, but should work most of the time for most - # of the formats, with not too many false positives. - # We don't have time to parse output much here. - res = res.replace('?ln=' + CFG_SITE_LANG, '?ln=' + ln) - res = res.replace('&ln=' + CFG_SITE_LANG, '&ln=' + ln) - res = res.replace('&ln=' + CFG_SITE_LANG, '&ln=' + ln) - out += res - return out, needs_2nd_pass - else: - if verbose == 9: - out += """\n
- No preformatted output found for record %s. - """% recID - - - # Live formatting of records in all other cases - if verbose == 9: - out += """\n
- Formatting record %i on-the-fly. - """ % recID - try: out_, needs_2nd_pass = format_record(recID=recID, of=of, ln=ln, verbose=verbose, search_pattern=search_pattern, xml_record=xml_record, user_info=user_info, **kwargs) out += out_ if of.lower() in ('xm', 'xoaimarc'): out = filter_hidden_fields(out, user_info, force_filtering=of.lower()=='xoaimarc') - # We have spent time computing this format - # We want to save this effort if the format is cached - if save_missing and recID and ln == cfg['CFG_SITE_LANG'] \ - and of.lower() in cfg['CFG_BIBFORMAT_CACHED_FORMATS'] \ - and verbose == 0: - bibformat_dblayer.save_preformatted_record(recID, - of, - out, - needs_2nd_pass) - return out, needs_2nd_pass except Exception: current_app.logger.exception( "An error occured while formatting record {recid} in {of}".format( recid=recID, of=of )) raise -def format_record_2nd_pass(recID, template, ln=CFG_SITE_LANG, - search_pattern=None, xml_record=None, - user_info=None, of=None, verbose=0, **kwargs): - # Create light bfo object - bfo = BibFormatObject(recID, ln, search_pattern, xml_record, user_info, of) - # Translations - template = translate_template(template, ln) - # Format template - r, dummy = format_with_format_template(format_template_filename=None, - format_template_code=template, - bfo=bfo, - verbose=verbose, - extra_context=kwargs) - return r - - def decide_format_template(bfo, of): """ Returns the format template name that should be used for formatting given output format and L{BibFormatObject}. Look at of rules, and take the first matching one. If no rule matches, returns None To match we ignore lettercase and spaces before and after value of rule and value of record :param bfo: a L{BibFormatObject} :param of: the code of the output format to use @return: name of a format template """ output_format = get_output_format(of) for rule in output_format.get('rules', []): if rule['field'].startswith('00'): # Rule uses controlfield values = [bfo.control_field(rule['field']).strip()] #Remove spaces else: # Rule uses datafield values = bfo.fields(rule['field']) # loop over multiple occurences, but take the first match if len(values) > 0: for value in values: value = value.strip() #Remove spaces pattern = rule['value'].strip() #Remove spaces match_obj = re.match(pattern, value, re.IGNORECASE) if match_obj is not None and \ match_obj.end() == len(value): return rule['template'] template = output_format.get('default', '') if template != '': return template else: return None def translate_template(template, ln=CFG_SITE_LANG): _ = gettext_set_language(ln) def translate(match): """ Translate matching values """ word = match.group("word") translated_word = _(word) return translated_word filtered_template = filter_languages(template, ln) evaluated_format = TRANSLATION_PATTERN.sub(translate, filtered_template) return evaluated_format def format_with_format_template(format_template_filename, bfo, verbose=0, format_template_code=None, qid="", extra_context=None): """ Format a record given a format template. Returns a formatted version of the record represented by bfo, in the language specified in bfo, and with the specified format template. If format_template_code is provided, the template will not be loaded from format_template_filename (but format_template_filename will still be used to determine if bft or xsl transformation applies). This allows to preview format code without having to save file on disk. :param format_template_filename: the dilename of a format template :param bfo: the object containing parameters for the current formatting :param format_template_code: if not empty, use code as template instead of reading format_template_filename (used for previews) :param verbose: the level of verbosity from 0 to 9 (O: silent, 5: errors, 7: errors and warnings, 9: errors and warnings, stop if error (debug mode )) @return: formatted text """ if format_template_code is not None: format_content = str(format_template_code) elif not format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION): format_content = get_format_template(format_template_filename)['code'] - if format_template_filename is None or \ - format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION): - # .bft - warnings.warn( - "BFT template format ({0}) is deprecated. " - "Use Jinja2 templates.".format(format_template_filename), - RemovedInInvenio22Warning - ) - evaluated_format, needs_2nd_pass = eval_format_template_elements( - format_content, - bfo, - verbose) - if not needs_2nd_pass: - evaluated_format = translate_template(evaluated_format, bfo.lang) - - - elif format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION): + if format_template_filename.endswith("." + CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION): evaluated_format = '' #try: from functools import wraps from invenio.modules.records.api import \ create_record as new_create_record, \ get_record as new_get_record from flask_login import current_user from invenio.base.helpers import unicodifier def _format_record(recid, of='hb', user_info=current_user, *args, **kwargs): from invenio.modules.formatter import format_record return format_record(recid, of, user_info=user_info, *args, **kwargs) # Fixes unicode problems in Jinja2 templates. def encode_utf8(f): @wraps(f) def wrapper(*args, **kwds): return unicodifier(f(*args, **kwds)) return wrapper if bfo.xml_record is None: record = new_get_record(bfo.recID) else: record = new_create_record(bfo.xml_record, master_format='marc') bfo.recID = bfo.recID if bfo.recID else 0 record.__getitem__ = encode_utf8(record.__getitem__) record.get = encode_utf8(record.get) - ctx = TEMPLATE_CONTEXT_FUNCTIONS_CACHE.functions - if extra_context is not None: - ctx.update(extra_context) - evaluated_format = render_template_to_string( 'format/record/'+format_template_filename, recid=bfo.recID, record=record, format_record=_format_record, qid=qid, - bfo=bfo, **ctx).encode('utf-8') + bfo=bfo, **(extra_context or {})).encode('utf-8') needs_2nd_pass = False else: from invenio.modules.records.api import get_record as new_get_record #.xsl if bfo.xml_record: # bfo was initialized with a custom MARCXML xml_record = '\n' + \ record_xml_output(bfo.record) else: # Fetch MARCXML. On-the-fly xm if we are now formatting in xm xml_record = '\n' + \ new_get_record(bfo.recID).legacy_export_as_marc() # Transform MARCXML using stylesheet evaluated_format = xslt.format(xml_record, template_source=format_content).decode('utf-8') needs_2nd_pass = False return evaluated_format, needs_2nd_pass -def eval_format_template_elements(format_template, bfo, verbose=0): - """ - Evalutes the format elements of the given template and replace each element with its value. - - Prepare the format template content so that we can directly replace the marc code by their value. - This implies: - 1. Look for special tags - 2. replace special tags by their evaluation - - :param format_template: the format template code - :param bfo: the object containing parameters for the current formatting - :param verbose: the level of verbosity from 0 to 9 (O: silent, - 5: errors, 7: errors and warnings, - 9: errors and warnings, stop if error (debug mode )) - @return: tuple (result, errors) - """ - _ = gettext_set_language(bfo.lang) - status = {'no_cache': False} - - # First define insert_element_code(match), used in re.sub() function - def insert_element_code(match): - """ - Analyses 'match', interpret the corresponding code, and return the result of the evaluation. - - Called by substitution in 'eval_format_template_elements(...)' - - :param match: a match object corresponding to the special tag that must be interpreted - """ - - function_name = match.group("function_name") - - # Ignore lang tags the processing is done outside - if function_name == 'lang': - return match.group(0) - - try: - format_element = get_format_element(function_name, verbose) - except Exception as e: - register_exception(req=bfo.req) - format_element = None - if verbose >= 5: - return '' + \ - cgi.escape(str(e)).replace('\n', '
') + \ - '' - if format_element is None: - try: - raise InvenioBibFormatError( - _('Could not find format element named %(function_name)s.', - function_name=function_name)) - except InvenioBibFormatError as exc: - register_exception(req=bfo.req) - - if verbose >= 5: - return '' + \ - str(exc.message)+'' - else: - params = {} - # Look for function parameters given in format template code - all_params = match.group('params') - if all_params is not None: - function_params_iterator = pattern_function_params.finditer(all_params) - for param_match in function_params_iterator: - name = param_match.group('param') - value = param_match.group('value') - params[name] = value - - if params.get('no_cache') == '1': - result = match.group("function_name") - del params['no_cache'] - if params: - params_str = ' '.join('%s="%s"' % (k, v) for k, v in params.iteritems()) - result = "" % (result, params_str) - else: - result = "" % result - status['no_cache'] = True - else: - # Evaluate element with params and return (Do not return errors) - result, dummy = eval_format_element(format_element, - bfo, - params, - verbose) - return result - - # Substitute special tags in the format by our own text. - # Special tags have the form - fmt = pattern_tag.sub(insert_element_code, format_template) - return fmt, status['no_cache'] - - -def eval_format_element(format_element, bfo, parameters=None, verbose=0): - """ - Returns the result of the evaluation of the given format element - name, with given L{BibFormatObject} and parameters. Also returns - the errors of the evaluation. - - :param format_element: a format element structure as returned by get_format_element - :param bfo: a L{BibFormatObject} used for formatting - :param parameters: a dict of parameters to be used for formatting. Key is parameter and value is value of parameter - :param verbose: the level of verbosity from 0 to 9 (O: silent, - 5: errors, - 7: errors and warnings, - 9: errors and warnings, stop if error (debug mode )) - - @return: tuple (result, errors) - """ - if parameters is None: - parameters = {} - - errors = [] - #Load special values given as parameters - prefix = parameters.get('prefix', "") - suffix = parameters.get('suffix', "") - default_value = parameters.get('default', "") - escape = parameters.get('escape', "") - output_text = '' - - _ = gettext_set_language(bfo.lang) - - # 3 possible cases: - # a) format element file is found: we execute it - # b) format element file is not found, but exist in tag table (e.g. bfe_isbn) - # c) format element is totally unknown. Do nothing or report error - if format_element is not None and format_element['type'] == "python": - # a) We found an element with the tag name, of type "python" - # Prepare a dict 'params' to pass as parameter to 'format' - # function of element - params = {} - - # Look for parameters defined in format element - # Fill them with specified default values and values - # given as parameters. - # Also remember if the element overrides the 'escape' - # parameter - format_element_overrides_escape = False - for param in format_element['attrs']['params']: - name = param['name'] - default = param['default'] - params[name] = parameters.get(name, default) - if name == 'escape': - format_element_overrides_escape = True - - # Add BibFormatObject - params['bfo'] = bfo - - # Execute function with given parameters and return result. - function = format_element['code'] - _ = gettext_set_language(bfo.lang) - - try: - output_text = function(**params) - except Exception as e: - register_exception(req=bfo.req) - name = format_element['attrs']['name'] - try: - raise InvenioBibFormatError( - _('Error when evaluating format element %(x_name)s ' - 'with parameters %(x_params)s.', - x_name=name, x_params=str(params))) - except InvenioBibFormatError, exc: - errors.append(exc.message) - - # Log exception - message = _( - "Error when evaluating format element %(format_element)s with" - " parameters %(parameters)s.", - format_element=name, - parameters=str(params) - ) - current_app.logger.exception( - message - ) - errors.append(message) - - # In debug mode - include traceback in output - if current_app.debug: - tb = sys.exc_info()[2] - stack = traceback.format_exception( - Exception, e, tb, limit=None - ) - output_text = '
' \
-                    '%s\n\n%s
' % (message, "".join(stack)) - - # None can be returned when evaluating function - if output_text is None: - output_text = "" - else: - try: - output_text = str(output_text) - except: - output_text = output_text.encode('utf-8') - - # Escaping: - # (1) By default, everything is escaped in mode 1 - # (2) If evaluated element has 'escape_values()' function, use - # its returned value as escape mode, and override (1) - # (3) If template has a defined parameter 'escape' (in allowed - # values), use it, and override (1) and (2). If this - # 'escape' parameter is overriden by the format element - # (defined in the 'format' function of the element), leave - # the escaping job to this element - - # (1) - escape_mode = 1 - - # (2) - escape_function = format_element['escape_function'] - if escape_function is not None: - try: - escape_mode = escape_function(bfo=bfo) - except Exception as e: - try: - raise InvenioBibFormatError(_('Escape mode for format element %(x_name)s could not be retrieved. Using default mode instead.', x_name=name)) - except InvenioBibFormatError as exc: - register_exception(req=bfo.req) - errors.append(exc.message) - - if verbose >= 5: - tb = sys.exc_info()[2] - - output_text += ''+ \ - str(exc.message) +' ' - # (3) - if escape in ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']: - escape_mode = int(escape) - - # If escape is equal to 1, then escape all - # HTML reserved chars. - if escape_mode > 0 and not format_element_overrides_escape: - output_text = escape_field(output_text, mode=escape_mode) - - # Add prefix and suffix if they have been given as parameters and if - # the evaluation of element is not empty - if output_text.strip() != "": - output_text = prefix + output_text + suffix - - # Add the default value if output_text is empty - if output_text == "": - output_text = default_value - - return output_text, errors - - elif format_element is not None and format_element['type'] == "field": - # b) We have not found an element in files that has the tag - # name. Then look for it in the table "tag" - # - # - # - - # Load special values given as parameters - separator = parameters.get('separator ', "") - nbMax = parameters.get('nbMax', "") - escape = parameters.get('escape', "1") # By default, escape here - - # Get the fields tags that have to be printed - tags = format_element['attrs']['tags'] - - output_text = [] - - # Get values corresponding to tags - for tag in tags: - p_tag = parse_tag(tag) - values = record_get_field_values(bfo.get_record(), - p_tag[0], - p_tag[1], - p_tag[2], - p_tag[3]) - if len(values)>0 and isinstance(values[0], dict): - #flatten dict to its values only - values_list = [x.values() for x in values] - #output_text.extend(values) - for values in values_list: - output_text.extend(values) - else: - output_text.extend(values) - - if nbMax != "": - try: - nbMax = int(nbMax) - except ValueError: - name = format_element['attrs']['name'] - try: - raise InvenioBibFormatError(_('"nbMax" parameter for %(x_param)s must be an "int".', x_param=name)) - except InvenioBibFormatError as exc: - register_exception(req=bfo.req) - errors.append(exc.message) - - if verbose >= 5: - output_text = output_text.append(exc.message) - else: - output_text = output_text[:nbMax] - - # Add prefix and suffix if they have been given as parameters and if - # the evaluation of element is not empty. - # If evaluation is empty string, return default value if it exists. - # Else return empty string - if ("".join(output_text)).strip() != "": - # If escape is equal to 1, then escape all - # HTML reserved chars. - if escape == '1': - output_text = cgi.escape(separator.join(output_text)) - else: - output_text = separator.join(output_text) - - output_text = prefix + output_text + suffix - else: - #Return default value - output_text = default_value - - return (output_text, errors) - else: - # c) Element is unknown - try: - raise InvenioBibFormatError(_('Could not find format element named %(format_element)s.', format_element=format_element)) - except InvenioBibFormatError as exc: - register_exception(req=bfo.req) - errors.append(exc.message) - - if verbose < 5: - return ("", errors) - elif verbose >= 5: - if verbose >= 9: - sys.exit(exc.message) - return ('' + - str(exc.message)+'', errors) - - def filter_languages(format_template, ln=CFG_SITE_LANG): """ Filters the language tags that do not correspond to the specified language. :param format_template: the format template code :param ln: the language that is NOT filtered out from the template @return: the format template with unnecessary languages filtered out """ # First define search_lang_tag(match) and clean_language_tag(match), used # in re.sub() function def search_lang_tag(match): """ Searches for the ... tag and remove inner localized tags such as , , that are not current_lang. If current_lang cannot be found inside ... , try to use 'CFG_SITE_LANG' :param match: a match object corresponding to the special tag that must be interpreted """ current_lang = ln def clean_language_tag(match): """ Return tag text content if tag language of match is output language. Called by substitution in 'filter_languages(...)' :param match: a match object corresponding to the special tag that must be interpreted """ if match.group(1) == current_lang: return match.group(2) else: return "" # End of clean_language_tag lang_tag_content = match.group("langs") # Try to find tag with current lang. If it does not exists, # then current_lang becomes CFG_SITE_LANG until the end of this # replace pattern_current_lang = re.compile(r"<(" + current_lang + r")\s*>(.*)()", re.IGNORECASE | re.DOTALL) if re.search(pattern_current_lang, lang_tag_content) is None: current_lang = CFG_SITE_LANG cleaned_lang_tag = ln_pattern.sub(clean_language_tag, lang_tag_content) return cleaned_lang_tag.strip() # End of search_lang_tag filtered_format_template = pattern_lang.sub(search_lang_tag, format_template) return filtered_format_template + def get_format_template(filename, with_attributes=False): """ Returns the structured content of the given formate template. if 'with_attributes' is true, returns the name and description. Else 'attrs' is not returned as key in dictionary (it might, if it has already been loaded previously):: {'code':"Some template code" 'attrs': {'name': "a name", 'description': "a description"} } :param filename: the filename of an format template :param with_attributes: if True, fetch the attributes (names and description) for format' @return: strucured content of format template """ - _ = gettext_set_language(CFG_SITE_LANG) - - if not filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION) and \ - not filename.endswith(".xsl"): - return None - - # Get from cache whenever possible - if filename in format_templates_cache: - # If we must return with attributes and template exist in - # cache with attributes then return cache. - # Else reload with attributes - if with_attributes and \ - 'attrs' in format_templates_cache[filename]: - return format_templates_cache[filename] + if not filename.lower().endswith(".xsl"): + raise RuntimeError('Unsupported file type {}.'.format(filename)) format_template = {'code': ""} try: path = registry.format_templates_lookup[filename] format_file = open(path) - format_content = format_file.read() + format_template['code'] = format_file.read() format_file.close() - - # Load format template code - # Remove name and description - if filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION): - code_and_description = pattern_format_template_name.sub("", - format_content, 1) - code = pattern_format_template_desc.sub("", code_and_description, 1) - else: - code = format_content - - format_template['code'] = code - - except: + except Exception: register_exception() - # Save attributes if necessary - if with_attributes: - format_template['attrs'] = get_format_template_attrs(filename) - - # Cache and return - format_templates_cache[filename] = format_template return format_template -def get_format_templates(with_attributes=False): - """ - Returns the list of all format templates, as dictionary with filenames as keys - - if 'with_attributes' is true, returns the name and description. Else 'attrs' is not - returned as key in each dictionary (it might, if it has already been loaded previously):: - - [{'code':"Some template code" - 'attrs': {'name': "a name", 'description': "a description"} - }, - ... - } - - :param with_attributes: if True, fetch the attributes (names and description) for formats - @return: the list of format templates (with code and info) - """ - format_templates = {} - - for filename in registry.format_templates: - if filename.endswith("."+CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION) or \ - filename.endswith(".xsl"): - filename = os.path.basename(filename) - format_templates[filename] = get_format_template(filename, - with_attributes) - - return format_templates - -def get_format_template_attrs(filename): - """ - Returns the attributes of the format template with given filename - - The attributes are {'name', 'description'} - Caution: the function does not check that path exists or - that the format element is valid. - :param filename: the name of a format template - @return: a structure with detailed information about given format template - """ - _ = gettext_set_language(CFG_SITE_LANG) - attrs = {} - attrs['name'] = "" - attrs['description'] = "" - try: - template_file = open(registry.format_templates_lookup[filename]) - code = template_file.read() - template_file.close() - - match = None - if filename.endswith(".xsl"): - # .xsl - attrs['name'] = filename[:-4] - else: - # .bft - match = pattern_format_template_name.search(code) - if match is not None: - attrs['name'] = match.group('name') - else: - attrs['name'] = filename - - - match = pattern_format_template_desc.search(code) - if match is not None: - attrs['description'] = match.group('desc').rstrip('.') - except Exception as e: - try: - raise InvenioBibFormatError(_('Could not read format template named %(filename)s. %(error)s.', filename=filename, error=str(e))) - except InvenioBibFormatError: - register_exception() - - attrs['name'] = filename - - return attrs - - -def get_format_element(element_name, verbose=0, with_built_in_params=False, - soft_fail=False): - """ - Returns the format element structured content. - - Return None if element cannot be loaded (file not found, not readable or - invalid) - - The returned structure is:: - {'attrs': {some attributes in dict. See get_format_element_attrs_from_*} - 'code': the_function_code, - 'type':"field" or "python" depending if element is defined in file or table, - 'escape_function': the function to call to know if element output must be escaped} - - :param element_name: the name of the format element to load - :param verbose: the level of verbosity from 0 to 9 (O: silent, - 5: errors, - 7: errors and warnings, - 9: errors and warnings, stop if error (debug mode )) - :param with_built_in_params: if True, load the parameters built in all elements - @return: a dictionary with format element attributes - """ - _ = gettext_set_language(CFG_SITE_LANG) - - # Resolve filename and prepare 'name' as key for the cache - filename = resolve_format_element_filename(element_name) - if filename is not None: - name = filename.upper() - else: - name = element_name.upper() - - if name in format_elements_cache: - element = format_elements_cache[name] - if not with_built_in_params or \ - (with_built_in_params and - 'builtin_params' in element['attrs']): - return element - - if filename is None: - # Element is maybe in tag table - if bibformat_dblayer.tag_exists_for_name(element_name): - format_element = {'attrs': get_format_element_attrs_from_table( - element_name, - with_built_in_params), - 'code': None, - 'escape_function': None, - 'type': "field"} - # Cache and returns - format_elements_cache[name] = format_element - return format_element - - elif soft_fail: - register_exception() - return None - else: - raise InvenioBibFormatError( - _('Format element %(x_element_name)s could not be found.', - x_element_name=element_name)) - - else: - format_element = {} - - module_name = filename - if module_name.endswith(".py"): - module_name = module_name[:-3] - - # Load function 'format_element()' inside element - try: - function_format = TEMPLATE_CONTEXT_FUNCTIONS_CACHE.\ - bibformat_elements()[module_name] - format_element['code'] = function_format - except: - if soft_fail: - register_exception() - return None - else: - raise - - # Load function 'escape_values()' inside element - format_element['escape_function'] = function_format._escape_values - - # Prepare, cache and return - format_element['attrs'] = get_format_element_attrs_from_function( - function_format, - element_name, - with_built_in_params) - format_element['type'] = "python" - format_elements_cache[name] = format_element - return format_element - -def get_format_elements(with_built_in_params=False): - """ - Returns the list of format elements attributes as dictionary structure - - Elements declared in files have priority over element declared in 'tag' table - The returned object has this format:: - {element_name1: {'attrs': {'description':..., 'seealso':... - 'params':[{'name':..., 'default':..., 'description':...}, ...] - 'builtin_params':[{'name':..., 'default':..., 'description':...}, ...] - }, - 'code': code_of_the_element - }, - element_name2: {...}, - ...} - - Returns only elements that could be loaded (not error in code) - - @return: a dict of format elements with name as key, and a dict as attributes - :param with_built_in_params: if True, load the parameters built in all elements - """ - format_elements = {} - - mappings = bibformat_dblayer.get_all_name_tag_mappings() - - for name in mappings: - format_elements[name.upper().replace(" ", "_").strip()] = get_format_element(name, with_built_in_params=with_built_in_params) - - for module in registry.format_elements: - filename = os.path.basename(module.__file__) - filename_test = filename.upper().replace(" ", "_") - if filename_test.endswith(".PYC"): - filename_test = filename_test[:-1] - if filename_test.endswith(".PY") and not filename.upper().startswith("__INIT__.PY"): - if filename_test.startswith("BFE_"): - filename_test = filename_test[4:] - element_name = filename_test[:-3] - element = get_format_element(element_name, - with_built_in_params=with_built_in_params, - soft_fail=True) - if element is not None: - format_elements[element_name] = element - - return format_elements - -def get_format_element_attrs_from_function(function, element_name, - with_built_in_params=False): - """ - Returns the attributes of the function given as parameter. - - It looks for standard parameters of the function, default - values and comments in the docstring. - - The attributes are:: - {'name' : "name of element" #basically the name of 'name' parameter - 'description': "a string description of the element", - 'seealso' : ["element_1.py", "element_2.py", ...] #a list of related elements - 'params': [{'name':"param_name", #a list of parameters for this element (except 'bfo') - 'default':"default value", - 'description': "a description"}, ...], - 'builtin_params': {name: {'name':"param_name",#the parameters builtin for all elem of this kind - 'default':"default value", - 'description': "a description"}, ...}, - } - :param function: the formatting function of a format element - :param element_name: the name of the element - :param with_built_in_params: if True, load the parameters built in all elements - @return: a structure with detailed information of a function - """ - - attrs = {} - attrs['description'] = "" - attrs['name'] = element_name.replace(" ", "_").upper() - attrs['seealso'] = [] - - docstring = function.__doc__ - if isinstance(docstring, str): - # Look for function description in docstring - #match = pattern_format_element_desc.search(docstring) - description = docstring.split("@param")[0] - description = description.split("@see:")[0] - attrs['description'] = description.strip().rstrip('.') - - # Look for @see: in docstring - match = pattern_format_element_seealso.search(docstring) - if match is not None: - elements = match.group('see').rstrip('.').split(",") - for element in elements: - attrs['seealso'].append(element.strip()) - - params = {} - # Look for parameters in function definition - args, dummy_varargs, dummy_varkw, defaults = inspect.getargspec(function) - - # Prepare args and defaults_list such that we can have a mapping - # from args to defaults - args.reverse() - if defaults is not None: - defaults_list = list(defaults) - defaults_list.reverse() - else: - defaults_list = [] - - for arg, default in map(None, args, defaults_list): - if arg == "bfo": - #Don't keep this as parameter. It is hidden to users, and - #exists in all elements of this kind - continue - param = {} - param['name'] = arg - if default is None: - #In case no check is made inside element, we prefer to - #print "" (nothing) than None in output - param['default'] = "" - else: - param['default'] = default - param['description'] = "(no description provided)" - - params[arg] = param - - if isinstance(docstring, str): - # Look for AT param descriptions in docstring. - # Add description to existing parameters in params dict - params_iterator = pattern_format_element_params.finditer(docstring) - for match in params_iterator: - name = match.group('name') - if name in params: - params[name]['description'] = match.group('desc').rstrip('.') - - attrs['params'] = params.values() - - # Load built-in parameters if necessary - if with_built_in_params: - - builtin_params = [] - # Add 'prefix' parameter - param_prefix = {} - param_prefix['name'] = "prefix" - param_prefix['default'] = "" - param_prefix['description'] = """A prefix printed only if the - record has a value for this element""" - builtin_params.append(param_prefix) - - # Add 'suffix' parameter - param_suffix = {} - param_suffix['name'] = "suffix" - param_suffix['default'] = "" - param_suffix['description'] = """A suffix printed only if the - record has a value for this element""" - builtin_params.append(param_suffix) - - # Add 'default' parameter - param_default = {} - param_default['name'] = "default" - param_default['default'] = "" - param_default['description'] = """A default value printed if the - record has no value for this element""" - builtin_params.append(param_default) - - # Add 'escape' parameter - param_escape = {} - param_escape['name'] = "escape" - param_escape['default'] = "" - param_escape['description'] = """0 keeps value as it is. Refer to main - documentation for escaping modes - 1 to 7""" - builtin_params.append(param_escape) - - attrs['builtin_params'] = builtin_params - - return attrs - -def get_format_element_attrs_from_table(element_name, - with_built_in_params=False): - """ - Returns the attributes of the format element with given name in 'tag' table. - - Returns None if element_name does not exist in tag table. - - The attributes are:: - {'name' : "name of element" #basically the name of 'element_name' parameter - 'description': "a string description of the element", - 'seealso' : [] #a list of related elements. Always empty in this case - 'params': [], #a list of parameters for this element. Always empty in this case - 'builtin_params': [{'name':"param_name", #the parameters builtin for all elem of this kind - 'default':"default value", - 'description': "a description"}, ...], - 'tags':["950.1", 203.a] #the list of tags printed by this element - } - - :param element_name: an element name in database - :param element_name: the name of the element - :param with_built_in_params: if True, load the parameters built in all elements - @return: a structure with detailed information of an element found in DB - """ - - attrs = {} - tags = bibformat_dblayer.get_tags_from_name(element_name) - field_label = "field" - if len(tags)>1: - field_label = "fields" - - attrs['description'] = "Prints %s %s of the record" % (field_label, - ", ".join(tags)) - attrs['name'] = element_name.replace(" ", "_").upper() - attrs['seealso'] = [] - attrs['params'] = [] - attrs['tags'] = tags - - # Load built-in parameters if necessary - if with_built_in_params: - builtin_params = [] - - # Add 'prefix' parameter - param_prefix = {} - param_prefix['name'] = "prefix" - param_prefix['default'] = "" - param_prefix['description'] = """A prefix printed only if the - record has a value for this element""" - builtin_params.append(param_prefix) - - # Add 'suffix' parameter - param_suffix = {} - param_suffix['name'] = "suffix" - param_suffix['default'] = "" - param_suffix['description'] = """A suffix printed only if the - record has a value for this element""" - builtin_params.append(param_suffix) - - # Add 'separator' parameter - param_separator = {} - param_separator['name'] = "separator" - param_separator['default'] = " " - param_separator['description'] = """A separator between elements of - the field""" - builtin_params.append(param_separator) - - # Add 'nbMax' parameter - param_nbMax = {} - param_nbMax['name'] = "nbMax" - param_nbMax['default'] = "" - param_nbMax['description'] = """The maximum number of values to - print for this element. No limit if not - specified""" - builtin_params.append(param_nbMax) - - # Add 'default' parameter - param_default = {} - param_default['name'] = "default" - param_default['default'] = "" - param_default['description'] = """A default value printed if the - record has no value for this element""" - builtin_params.append(param_default) - - # Add 'escape' parameter - param_escape = {} - param_escape['name'] = "escape" - param_escape['default'] = "" - param_escape['description'] = """If set to 1, replaces special - characters '&', '<' and '>' of this - element by SGML entities""" - builtin_params.append(param_escape) - - attrs['builtin_params'] = builtin_params - - return attrs - - def get_output_format(code, with_attributes=False, verbose=0): """Return the structured content of the given output format.""" try: return registry.output_formats[code.lower()] except KeyError: raise InvenioBibFormatError("Missing output format '{0}'".format(code)) def get_output_formats(with_attributes=False): """Return all output format as a dictionary with their code as key.""" assert with_attributes == False return dict(registry.output_formats) -def resolve_format_element_filename(element_name): - """ - Returns the filename of element corresponding to x{element_name} - - This is necessary since format templates code call - elements by ignoring case, for eg. is the - same as . - It is also recommended that format elements filenames are - prefixed with bfe_ . We need to look for these too. - - The name of the element has to start with "BFE_". - - :param element_name: a name for a format element - @return: the corresponding filename, with right case - """ - - if not element_name.endswith(".py"): - name = element_name.replace(" ", "_").upper() +".PY" - else: - name = element_name.replace(" ", "_").upper() - - files = registry.format_elements - - for element in files: - filename = element.__file__ - if filename.endswith('.pyc'): - filename = filename[:-1] - basename = os.path.basename(filename) - - test_filename = basename.replace(" ", "_").upper() - - if test_filename == name or \ - test_filename == "BFE_" + name or \ - "BFE_" + test_filename == name: - return basename - - # No element with that name found - # Do not log error, as it might be a normal execution case: - # element can be in database - return None - - def clear_caches(): """ Clear the caches (Output Format, Format Templates and Format Elements). @return: None """ - global format_templates_cache, format_elements_cache, format_outputs_cache + global format_templates_cache, format_outputs_cache format_templates_cache = {} - format_elements_cache = {} format_outputs_cache = {} class BibFormatObject(object): """ An object that encapsulates a record and associated methods, and that is given as parameter to all format elements 'format' function. The object is made specifically for a given formatting, i.e. it includes for example the language for the formatting. The object provides basic accessors to the record. For full access, one can get the record with get_record() and then use BibRecord methods on the returned object. """ # The record record = None # The language in which the formatting has to be done lang = CFG_SITE_LANG # A list of string describing the context in which the record has # to be formatted. # It represents the words of the user request in web interface search search_pattern = [] # The id of the record recID = 0 # The information about the user, as returned by # 'webuser.collect_user_info(req)' user_info = None # The format in which the record is being formatted output_format = '' req = None # DEPRECATED: use bfo.user_info instead. Used by WebJournal. def __init__(self, recID, ln=CFG_SITE_LANG, search_pattern=None, xml_record=None, user_info=None, output_format=''): """ Creates a new bibformat object, with given record. You can either specify an record ID to format, or give its xml representation. if 'xml_record' is not None, use 'xml_record' instead of recID for the record. 'user_info' allows to grant access to some functionalities on a page depending on the user's priviledges. It is a dictionary in the following form:: user_info = { 'remote_ip' : '', 'remote_host' : '', 'referer' : '', 'uri' : '', 'agent' : '', 'uid' : -1, 'nickname' : '', 'email' : '', 'group' : [], 'guest' : '1' } :param recID: the id of a record :param ln: the language in which the record has to be formatted :param search_pattern: list of string representing the request used by the user in web interface :param xml_record: a xml string of the record to format :param user_info: the information of the user who will view the formatted page :param output_format: the output_format used for formatting this record """ self.xml_record = None # *Must* remain empty if recid is given if xml_record is not None: # If record is given as parameter self.xml_record = xml_record self.record = create_record(xml_record)[0] recID = record_get_field_value(self.record, "001") or None recID = int(recID) if recID is not None else recID try: assert isinstance(recID, (int, long, type(None))), 'Argument of wrong type!' except AssertionError: register_exception(prefix="recid needs to be an integer in BibFormatObject", alert_admin=True) recID = int(recID) self.recID = recID self.lang = wash_language(ln) if search_pattern is None: search_pattern = [] self.search_pattern = search_pattern self.output_format = output_format self.user_info = user_info if self.user_info is None: from invenio.ext.login.legacy_user import UserInfo self.user_info = UserInfo(None) def get_record(self): """ Returns the record structure of this L{BibFormatObject} instance @return: the record structure as defined by BibRecord library """ from invenio.legacy.search_engine import get_record # Create record if necessary if self.record is None: # on-the-fly creation if current output is xm self.record = get_record(self.recID) return self.record def control_field(self, tag, escape=0): """ Returns the value of control field given by tag in record :param tag: the marc code of a field :param escape: 1 if returned value should be escaped. Else 0. @return: value of field tag in record """ if self.get_record() is None: #Case where BibRecord could not parse object return '' p_tag = parse_tag(tag) field_value = record_get_field_value(self.get_record(), p_tag[0], p_tag[1], p_tag[2], p_tag[3]) if escape == 0: return field_value else: return escape_field(field_value, escape) def field(self, tag, escape=0): """ Returns the value of the field corresponding to tag in the current record. If the value does not exist, return empty string. Else returns the same as bfo.fields(..)[0] (see docstring below). 'escape' parameter allows to escape special characters of the field. The value of escape can be: 0. no escaping 1. escape all HTML characters 2. remove unsafe HTML tags (Eg. keep
) 3. Mix of mode 1 and 2. If value of field starts with , then use mode 2. Else use mode 1. 4. Remove all HTML tags 5. Same as 2, with more tags allowed (like ) 6. Same as 3, with more tags allowed (like ) 7. Mix of mode 0 and mode 1. If field_value starts with , then use mode 0. Else use mode 1. 8. Same as mode 1, but also escape double-quotes 9. Same as mode 4, but also escape double-quotes :param tag: the marc code of a field :param escape: 1 if returned value should be escaped. Else 0. (see above for other modes) @return: value of field tag in record """ list_of_fields = self.fields(tag) if len(list_of_fields) > 0: # Escaping below if escape == 0: return list_of_fields[0] else: return escape_field(list_of_fields[0], escape) else: return "" def fields(self, tag, escape=0, repeatable_subfields_p=False): """ Returns the list of values corresonding to "tag". If tag has an undefined subcode (such as 999C5), the function returns a list of dictionaries, whoose keys are the subcodes and the values are the values of tag.subcode. If the tag has a subcode, simply returns list of values corresponding to tag. Eg. for given MARC:: 999C5 $a value_1a $b value_1b 999C5 $b value_2b 999C5 $b value_3b $b value_3b_bis >>> bfo.fields('999C5b') >>> ['value_1b', 'value_2b', 'value_3b', 'value_3b_bis'] >>> bfo.fields('999C5') >>> [{'a':'value_1a', 'b':'value_1b'}, {'b':'value_2b'}, {'b':'value_3b'}] By default the function returns only one value for each subfield (that is it considers that repeatable subfields are not allowed). It is why in the above example 'value3b_bis' is not shown for bfo.fields('999C5'). (Note that it is not defined which of value_3b or value_3b_bis is returned). This is to simplify the use of the function, as most of the time subfields are not repeatable (in that way we get a string instead of a list). You can allow repeatable subfields by setting 'repeatable_subfields_p' parameter to True. In this mode, the above example would return: >>> bfo.fields('999C5b', repeatable_subfields_p=True) >>> ['value_1b', 'value_2b', 'value_3b'] >>> bfo.fields('999C5', repeatable_subfields_p=True) >>> [{'a':['value_1a'], 'b':['value_1b']}, {'b':['value_2b']}, {'b':['value_3b', 'value3b_bis']}] NOTICE THAT THE RETURNED STRUCTURE IS DIFFERENT. Also note that whatever the value of 'repeatable_subfields_p' is, bfo.fields('999C5b') always show all fields, even repeatable ones. This is because the parameter has no impact on the returned structure (it is always a list). 'escape' parameter allows to escape special characters of the fields. The value of escape can be: 0. No escaping 1. Escape all HTML characters 2. Remove unsafe HTML tags (Eg. keep
) 3. Mix of mode 1 and 2. If value of field starts with , then use mode 2. Else use mode 1. 4. Remove all HTML tags 5. Same as 2, with more tags allowed (like ) 6. Same as 3, with more tags allowed (like ) 7. Mix of mode 0 and mode 1. If field_value starts with , then use mode 0. Else use mode 1. 8. Same as mode 1, but also escape double-quotes 9. Same as mode 4, but also escape double-quotes :param tag: the marc code of a field :param escape: 1 if returned values should be escaped. Else 0. @repeatable_subfields_p if True, returns the list of subfields in the dictionary @return: values of field tag in record """ if self.get_record() is None: # Case where BibRecord could not parse object return [] p_tag = parse_tag(tag) if p_tag[3] != "": # Subcode has been defined. Simply returns list of values values = record_get_field_values(self.get_record(), p_tag[0], p_tag[1], p_tag[2], p_tag[3]) if escape == 0: return values else: return [escape_field(value, escape) for value in values] else: # Subcode is undefined. Returns list of dicts. # However it might be the case of a control field. instances = record_get_field_instances(self.get_record(), p_tag[0], p_tag[1], p_tag[2]) if repeatable_subfields_p: list_of_instances = [] for instance in instances: instance_dict = {} for subfield in instance[0]: if subfield[0] not in instance_dict: instance_dict[subfield[0]] = [] if escape == 0: instance_dict[subfield[0]].append(subfield[1]) else: instance_dict[subfield[0]].append(escape_field(subfield[1], escape)) list_of_instances.append(instance_dict) return list_of_instances else: if escape == 0: return [dict(instance[0]) for instance in instances] else: return [dict([(subfield[0], escape_field(subfield[1], escape)) for subfield in instance[0]]) for instance in instances] def kb(self, kb, string, default=""): """ Returns the value of the "string" in the knowledge base "kb". If kb does not exist or string does not exist in kb, returns 'default' string or empty string if not specified. :param kb: a knowledge base name :param string: the string we want to translate :param default: a default value returned if 'string' not found in 'kb' @return: a string value corresponding to translated input with given kb """ if not string: return default val = get_kbr_values(kb, searchkey=string, searchtype='e') try: return val[0][0] except IndexError: return default # Utility functions # def escape_field(value, mode=0): """ Utility function used to escape the value of a field in given mode. - mode 0: no escaping - mode 1: escaping all HTML/XML characters (escaped chars are shown as escaped) - mode 2: escaping unsafe HTML tags to avoid XSS, but keep basic one (such as
) Escaped tags are removed. - mode 3: mix of mode 1 and mode 2. If field_value starts with , then use mode 2. Else use mode 1. - mode 4: escaping all HTML/XML tags (escaped tags are removed) - mode 5: same as 2, but allows more tags, like - mode 6: same as 3, but allows more tags, like - mode 7: mix of mode 0 and mode 1. If field_value starts with , then use mode 0. Else use mode 1. - mode 8: same as mode 1, but also escape double-quotes - mode 9: same as mode 4, but also escape double-quotes :param value: value to escape :param mode: escaping mode to use @return: an escaped version of X{value} according to chosen X{mode} """ if mode == 1: return cgi.escape(value) elif mode == 8: return cgi.escape(value, True) elif mode in [2, 5]: allowed_attribute_whitelist = CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST allowed_tag_whitelist = CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST + \ ('class',) if mode == 5: allowed_attribute_whitelist += ('src', 'alt', 'width', 'height', 'style', 'summary', 'border', 'cellspacing', 'cellpadding') allowed_tag_whitelist += ('img', 'table', 'td', 'tr', 'th', 'span', 'caption') try: return washer.wash(value, allowed_attribute_whitelist= allowed_attribute_whitelist, allowed_tag_whitelist= allowed_tag_whitelist ) except HTMLParseError: # Parsing failed return cgi.escape(value) elif mode in [3, 6]: if value.lstrip(' \n').startswith(html_field): allowed_attribute_whitelist = CFG_HTML_BUFFER_ALLOWED_ATTRIBUTE_WHITELIST allowed_tag_whitelist = CFG_HTML_BUFFER_ALLOWED_TAG_WHITELIST + \ ('class',) if mode == 6: allowed_attribute_whitelist += ('src', 'alt', 'width', 'height', 'style', 'summary', 'border', 'cellspacing', 'cellpadding') allowed_tag_whitelist += ('img', 'table', 'td', 'tr', 'th', 'span', 'caption') try: return washer.wash(value, allowed_attribute_whitelist= allowed_attribute_whitelist, allowed_tag_whitelist= allowed_tag_whitelist ) except HTMLParseError: # Parsing failed return cgi.escape(value) else: return cgi.escape(value) elif mode in [4, 9]: try: out = washer.wash(value, allowed_attribute_whitelist=[], allowed_tag_whitelist=[] ) if mode == 9: out = out.replace('"', '"') return out except HTMLParseError: # Parsing failed if mode == 4: return cgi.escape(value) else: return cgi.escape(value, True) elif mode == 7: if value.lstrip(' \n').startswith(html_field): return value else: return cgi.escape(value) else: return value def make_filter_line(hide_tag): """Generate a line used for filtering MARCXML.""" hide_tag = str(hide_tag) tag = hide_tag[:3] ind1 = hide_tag[3:4] ind2 = hide_tag[4:5] if ind1 == "_": ind1 = " " if ind2 == "_": ind2 = " " if not ind1 and not ind2: return 'datafield tag="%s"' % tag if not ind2 and ind1: return 'datafield tag="%s" ind1="%s"' % (tag, ind1) return 'datafield tag="%s" ind1="%s" ind2="%s"' % (tag, ind1, ind2) def filter_hidden_fields(recxml, user_info=None, filter_tags=None, force_filtering=False): """ Filter out tags specified by filter_tags from MARCXML. If the user is allowed to run bibedit, then filter nothing, unless force_filtering is set to True. :param recxml: marcxml presentation of the record :param user_info: user information; if None, then assume invoked via CLI with all rights :param filter_tags: list of MARC tags to be filtered :param force_filtering: do we force filtering regardless of user rights? :return: recxml without the hidden fields """ filter_tags = filter_tags or cfg['CFG_BIBFORMAT_HIDDEN_TAGS'] if force_filtering: pass else: if user_info is None: #by default return recxml else: if (acc_authorize_action(user_info, 'runbibedit')[0] == 0): #no need to filter return recxml #filter.. out = "" omit = False filter_lines = map(make_filter_line, filter_tags) for line in recxml.splitlines(True): #check if this block needs to be omitted for htag in filter_lines: if htag in line: omit = True if not omit: out += line if omit and ('' in line or '' in line): omit = False return out diff --git a/invenio/modules/formatter/format_elements/__init__.py b/invenio/modules/formatter/format_elements/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/invenio/modules/formatter/format_elements/bfe_abstract.py b/invenio/modules/formatter/format_elements/bfe_abstract.py deleted file mode 100644 index 1b9cefbbc..000000000 --- a/invenio/modules/formatter/format_elements/bfe_abstract.py +++ /dev/null @@ -1,173 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints English and French abstract. -""" - -__revision__ = "$Id$" - -#import cgi -from invenio.modules.formatter import utils as bibformat_utils - -def format_element(bfo, prefix_en, prefix_fr, suffix_en, suffix_fr, limit, max_chars, - extension_en="[...] ",extension_fr="[...] ", contextual="no", - highlight='no', print_lang='en,fr', escape="3", - separator_en="
", separator_fr="
", latex_to_html='no'): - """ Prints the abstract of a record in HTML. By default prints - English and French versions. - - Printed languages can be chosen with the 'print_lang' parameter. - - @param prefix_en: a prefix for english abstract (printed only if english abstract exists) - @param prefix_fr: a prefix for french abstract (printed only if french abstract exists) - @param limit: the maximum number of sentences of the abstract to display (for each language) - @param max_chars: the maximum number of chars of the abstract to display (for each language) - @param extension_en: a text printed after english abstracts longer than parameter 'limit' - @param extension_fr: a text printed after french abstracts longer than parameter 'limit' - @param suffix_en: a suffix for english abstract(printed only if english abstract exists) - @param suffix_fr: a suffix for french abstract(printed only if french abstract exists) - @parmm contextual if 'yes' prints sentences the most relative to user search keyword (if limit < abstract) - @param highlight: if 'yes' highlights words from user search keyword - @param print_lang: the comma-separated list of languages to print. Now restricted to 'en' and 'fr' - @param escape: escaping method (overrides default escape parameter to not escape separators) - @param separator_en: a separator between each english abstract - @param separator_fr: a separator between each french abstract - @param latex_to_html: if 'yes', interpret as LaTeX abstract - """ - out = '' - - if print_lang == 'auto': - print_lang = bfo.lang - languages = print_lang.split(',') - - try: - escape_mode_int = int(escape) - except ValueError as e: - escape_mode_int = 0 - - abstract_en = bfo.fields('520__a', escape=escape_mode_int) - abstract_en.extend(bfo.fields('520__b', escape=escape_mode_int)) - abstract_en = separator_en.join(abstract_en) - - abstract_fr = bfo.fields('590__a', escape=escape_mode_int) - abstract_fr.extend(bfo.fields('590__b', escape=escape_mode_int)) - abstract_fr = separator_fr.join(abstract_fr) - - if contextual == 'yes' and limit != "" and \ - limit.isdigit() and int(limit) > 0: - context_en = bibformat_utils.get_contextual_content(abstract_en, - bfo.search_pattern, - max_lines=int(limit)) - #FIXME add something like [...] before and after - #contextual sentences when not at beginning/end of abstract - #if not abstract_en.strip().startswith(context_en[0].strip()): - # out += '[...]' - abstract_en = "
".join(context_en) - - context_fr = bibformat_utils.get_contextual_content(abstract_fr, - bfo.search_pattern, - max_lines=int(limit)) - abstract_fr = "
".join(context_fr) - - if len(abstract_en) > 0 and 'en' in languages: - - out += prefix_en - print_extension = False - - if max_chars != "" and max_chars.isdigit() and \ - int(max_chars) < len(abstract_en): - print_extension = True - abstract_en = abstract_en[:int(max_chars)] - - if limit != "" and limit.isdigit(): - s_abstract = abstract_en.split(". ") # Split around - # DOTSPACE so that we - # don't split html - # links - - if int(limit) < len(s_abstract): - print_extension = True - s_abstract = s_abstract[:int(limit)] - - #for sentence in s_abstract: - # out += sentence + "." - out = '. '.join(s_abstract) - - # Add final dot if needed - if abstract_en.endswith('.'): - out += '.' - - if print_extension: - out += " " + extension_en - - else: - out += abstract_en - - out += suffix_en - - if len(abstract_fr) > 0 and 'fr' in languages: - - out += prefix_fr - - print_extension = False - - if max_chars != "" and max_chars.isdigit() and \ - int(max_chars) < len(abstract_fr): - print_extension = True - abstract_fr = abstract_fr[:int(max_chars)] - - if limit != "" and limit.isdigit(): - s_abstract = abstract_fr.split(". ") # Split around - # DOTSPACE so that we - # don't split html - # links - - if int(limit) < len(s_abstract): - print_extension = True - s_abstract = s_abstract[:int(limit)] - - #for sentence in s_abstract: - # out += sentence + "." - out += '. '.join(s_abstract) - - # Add final dot if needed - if abstract_fr.endswith('.'): - out += '.' - - if print_extension: - out += " "+extension_fr - - else: - out += abstract_fr - - out += suffix_fr - - if highlight == 'yes': - out = bibformat_utils.highlight(out, bfo.search_pattern) - - if latex_to_html == 'yes': - out = bibformat_utils.latex_to_html(out) - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_additional_report_numbers.py b/invenio/modules/formatter/format_elements/bfe_additional_report_numbers.py deleted file mode 100644 index f11897e77..000000000 --- a/invenio/modules/formatter/format_elements/bfe_additional_report_numbers.py +++ /dev/null @@ -1,49 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints additional report numbers -""" - -__revision__ = "$Id$" - -from invenio.modules.formatter.format_elements.bfe_report_numbers import \ - build_report_number_link - -def format_element(bfo, limit, separator=" ", link='yes'): - """ - Prints the additional report numbers of the record - - @param separator: the separator between report numbers. - @param limit: the max number of report numbers to display - @param link: if 'yes', display report number with corresponding link when possible - """ - numbers = bfo.fields("088__a") - - if limit.isdigit() and int(limit) <= len(numbers): - numbers = numbers[:int(limit)] - - return separator.join([build_report_number_link(report_number, - link == 'yes') \ - for report_number in numbers]) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_addresses.py b/invenio/modules/formatter/format_elements/bfe_addresses.py deleted file mode 100644 index e4b409151..000000000 --- a/invenio/modules/formatter/format_elements/bfe_addresses.py +++ /dev/null @@ -1,58 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints list of addresses -""" - -__revision__ = "$Id$" - -import cgi -from urllib import quote -from invenio.config import CFG_BASE_URL - -def format_element(bfo, separator="; ", print_link="yes"): - """ - Prints a list of addresses linked to this report - - @param separator: the separator between addresses. - @param print_link: Links the addresses to search engine (HTML links) if 'yes' - """ - - addresses = bfo.fields('270') - list_addresses = [] - if print_link.lower() == 'yes': - for address in addresses: - list_addresses.append(''+cgi.escape(address.get('p', "")) + \ - '') - list_addresses.append(cgi.escape(address.get('g', ""))) - else: - for address in addresses: - list_addresses.append(cgi.escape(address.get('p', ""))) - list_addresses.append(cgi.escape(address.get('g', ""))) - - return separator.join(list_addresses) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_addthis.py b/invenio/modules/formatter/format_elements/bfe_addthis.py deleted file mode 100644 index 641e466ef..000000000 --- a/invenio/modules/formatter/format_elements/bfe_addthis.py +++ /dev/null @@ -1,64 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - wrap the Add This service: .""" - -try: - from invenio.config import CFG_BIBFORMAT_ADDTHIS_ID -except ImportError: - CFG_BIBFORMAT_ADDTHIS_ID = None - -from invenio.legacy.search_engine import get_all_restricted_recids - - -def format_element(bfo, only_public_records=1, - addthis_id=CFG_BIBFORMAT_ADDTHIS_ID): - """Print the AddThis box from the service. - - :param only_public_records: if set to 1 (the default), prints the box only - if the record is public (i.e. if it belongs to the root colletion and - is accessible to the world). - :param addthis_id: the pubid API parameter as provided by the service - (e.g. ra-4ff80aae118f4dad). This can be set at the repository level - in the variable CFG_BIBFORMAT_ADDTHIS_ID in invenio(-local).conf - """ - if not addthis_id: - return "" - if int(only_public_records) and \ - bfo.recID not in get_all_restricted_recids(): - return "" - return """\ - -
- - - - - - -
- - - -""" % {'addthis_id': addthis_id} - - -def escape_values(bfo): - """Check if output of this element should be escaped.""" - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_affiliation.py b/invenio/modules/formatter/format_elements/bfe_affiliation.py deleted file mode 100644 index 08bdc96b7..000000000 --- a/invenio/modules/formatter/format_elements/bfe_affiliation.py +++ /dev/null @@ -1,71 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints affiliation -""" -__revision__ = "$Id$" - -import cgi -from invenio.config import \ - CFG_SITE_URL, CFG_SITE_NAME -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME - -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no - -def format_element(bfo): - """ - HTML Affiliation display - """ - affiliations = bfo.fields('909C1', repeatable_subfields_p=True) - out = "" - for affiliation_dict in affiliations: - if 'u' in affiliation_dict: - recIDs = [] - affiliation = affiliation_dict['u'][0] - control_nos = affiliation_dict.get('0') - for control_no in control_nos or []: - recIDs.extend(get_low_level_recIDs_from_control_no(control_no)) - affiliation = cgi.escape(affiliation) - if len(recIDs) == 1: - affiliation = '' + affiliation + '' - elif len(recIDs) > 1: - affiliation = '' + affiliation + '' - - out += affiliation + " " - - if out: - return "
" + out - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_aid_authors.py b/invenio/modules/formatter/format_elements/bfe_aid_authors.py deleted file mode 100644 index 16368ae35..000000000 --- a/invenio/modules/formatter/format_elements/bfe_aid_authors.py +++ /dev/null @@ -1,313 +0,0 @@ -# -*- coding: utf-8 -*- -# -# $Id$ -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints authors -""" - -def format_element(bfo, limit, separator='; ', - extension='[...]', - print_links="yes", - print_affiliations='no', - affiliation_prefix=' (', - affiliation_suffix=')', - print_affiliation_first='no', - interactive="no", - highlight="no", - affiliations_separator=" ; ", - name_last_first="yes", - collaboration="yes", - id_links="no", - markup="html", - link_extension="no", - ): - """ - Prints the list of authors of a record. - - @param limit: the maximum number of authors to display - @param separator: the separator between authors. - @param extension: a text printed if more authors than 'limit' exist - @param print_links: if yes, prints the authors as HTML link to their publications - @param print_affiliations: if yes, make each author name followed by its affiliation - @param affiliation_prefix: prefix printed before each affiliation - @param affiliation_suffix: suffix printed after each affiliation - @param print_affiliation_first: if 'yes', affiliation is printed before the author - @param interactive: if yes, enable user to show/hide authors when there are too many (html + javascript) - @param highlight: highlights authors corresponding to search query if set to 'yes' - @param affiliations_separator: separates affiliation groups - @param name_last_first: if yes (default) print last, first otherwise first last - @param collaboration: if yes (default) uses collaboration name in place of long author list, if available - @param id_links: if yes (default = no) prints link based on INSPIRE IDs if available - only used if print_links = yes - @param markup: html (default) or latex controls small markup differences - @param link_extension: if 'yes' link the extension to the detailed - record page - - """ - from urllib import quote - from cgi import escape - import re - from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD, CFG_SITE_URL - - #regex for parsing last and first names and initials - re_last_first = re.compile('^(?P[^,]+)\s*,\s*(?P[^\,]*)(?P\,?.*)$') - re_initials = re.compile(r'(?P\w)(\w+|\.)\s*') - re_coll = re.compile(r'\s*collaborations?', re.IGNORECASE) - - from invenio.base.i18n import gettext_set_language - - _ = gettext_set_language(bfo.lang) # load the right message language - - bibrec_id = bfo.control_field("001") - authors = [] - authors = bfo.fields('100__', repeatable_subfields_p=True) - authors.extend(bfo.fields('700__', repeatable_subfields_p=True)) - - # Keep real num of authorsfix + affiliations_separator.join(author['u']) + \ - nb_authors = len(authors) - - # Limit num of authors, so that we do not process - # the authors that will not be shown. This can only - # be done in non-interactive mode, as interactive mode - # allows to show all of them. - if limit.isdigit() and nb_authors > int(limit) \ - and interactive != "yes": - if bfo.field('710g'): #check for colln note - authors = authors[:1] - else: - authors = authors[:int(limit)] - - # Process authors to add link, affiliation and highlight - for author in authors: - - if 'a' in author: - author['a'] = author['a'][0] # There should not be - if highlight == 'yes': - from invenio.modules.formatter import utils as bibformat_utils - author['a'] = bibformat_utils.highlight(author['a'], - bfo.search_pattern) - - #check if we need to reverse last, first - #we don't try to reverse it if it isn't stored with a comma. - first_last_match = re_last_first.search(author['a']) - author['display'] = author['a'] - - if name_last_first.lower() == "no": - if first_last_match: - author['display'] = first_last_match.group('first_names') + \ - ' ' + \ - first_last_match.group('last') + \ - first_last_match.group('extension') - - #for latex we do initials only (asn assume first last) - if markup == 'latex': - if first_last_match: - first = re_initials.sub('\g.~', \ - first_last_match.group('first_names')) - author['display'] = first + \ - first_last_match.group('last') + \ - first_last_match.group('extension') - - - if print_links.lower() == "yes": - - # if there is an ID, search using that. - id_link = '' - if id_links == "yes" and 'i' in author: - author['i'] = author['i'][0] #possible to have more IDs? - id_link = '' + escape("(ID Search)") + ' ' - - - author['display'] = '' + escape(author['display']) + '' + \ - id_link - - if print_affiliations == "yes": - if 'e' in author: - author['e'] = affiliation_prefix + \ - affiliations_separator.join(author['e']) + \ - affiliation_suffix - - - - if 'u' in author: - author['ilink'] = ['%s' % string.lstrip() - for string in author['u']] - author['u'] = affiliation_prefix + \ - affiliations_separator.join(author['ilink']) + \ - affiliation_suffix - - - -# -# Consolidate repeated affiliations -# - - last = '' - authors.reverse() - for author in authors: - if 'u' not in author: - author['u'] = '' - #print 'this->'+ author['a']+'\n' - if last == author['u']: - author['u'] = '' - else: - last = author['u'] - - authors.reverse() - - # Flatten author instances - if print_affiliations == 'yes': -# 100__a (100__e) 700__a (100__e) (100__u) - if print_affiliation_first.lower() != 'yes': - authors = [author.get('display', '') + author.get('u', '') \ - for author in authors] - - else: - authors = [author.get('u', '') + author.get('display', '') \ - for author in authors] - - - else: - authors = [author.get('display', '') - for author in authors] - - # link the extension to detailed record - if link_extension == 'yes' and interactive != 'yes': - extension = '' + \ - extension + '' - - # Detect Collaborations: - if collaboration == "yes": - colls = bfo.fields("710__g") - else: - colls = [] - if colls: - short_coll = False - colls = [re_coll.sub('', coll) for coll in colls] - if print_links.lower() == "yes": - colls = ['' + escape(coll) + '' for coll in colls] - - coll_display = " and ".join(colls) - if not coll_display.endswith("aboration"): - coll_display += " Collaboration" - if len(colls) > 1: - coll_display += 's' - if nb_authors > 1: - if markup == 'latex': - coll_display = authors[0] + extension + " [ " + \ - coll_display + " ]" - elif interactive == "yes": - coll_display += " (" + authors[0] + " " - extension += ")" - else: #html - coll_display += " (" + authors[0] + extension + ")" - elif nb_authors == 1: - short_coll = True - if markup == 'latex': - coll_display = authors[0] + " [ " + coll_display + " ]" - else: #html - coll_display += " (" + authors[0] + " for the collaboration)" - elif nb_authors == 0: - short_coll = True - if markup == 'latex': - coll_display = "[ " + coll_display + " ]" - - - - # Start outputting, depending on options and number of authors - if colls and (interactive != "yes" or short_coll): - return coll_display - - if limit.isdigit() and nb_authors > int(limit) and interactive != "yes": - return separator.join(authors[:len(authors)]) + \ - extension - - - elif interactive == "yes" and ((colls and not short_coll) or (limit.isdigit() and nb_authors > int(limit))): - out = ''' - - ''' % {'show_less':_("Hide"), - 'show_more':_("Show all %(x_num)i authors", x_num=nb_authors), - 'extension':extension} - -# out += '' - if colls: - show = coll_display - more = separator + separator.join(authors[1:]) + ')' - else: - show = separator.join(authors[:int(limit)]) - more = separator.join(authors[int(limit):len(authors)]) - - out += show - out += ' ' + more + '' - out += ' ' - out += ' ' - out += '' - return out - elif nb_authors > 0: - return separator.join(authors) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_appears_in_collections.py b/invenio/modules/formatter/format_elements/bfe_appears_in_collections.py deleted file mode 100644 index be18d3ce6..000000000 --- a/invenio/modules/formatter/format_elements/bfe_appears_in_collections.py +++ /dev/null @@ -1,50 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints publisher name -""" -__revision__ = "$Id$" - -from invenio.legacy.search_engine import get_all_collections_of_a_record, \ - create_navtrail_links - -def format_element(bfo, separator="
"): - """Prints the list of collections the record belongs to. - - @param separator: a separator between each collection link. - """ - - coll_names = get_all_collections_of_a_record(bfo.recID) - navtrails = [create_navtrail_links(coll_name, ln=bfo.lang) for coll_name in coll_names] - navtrails = [navtrail for navtrail in navtrails if navtrail] - navtrails.sort(lambda x, y: cmp(len(y), len(x))) - final_navtrails = [] - for navtrail in navtrails: - for final_navtrail in final_navtrails: - if navtrail in final_navtrail: - break - else: - final_navtrails.append(navtrail) - return separator.join(final_navtrails) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_arxiv_link.py b/invenio/modules/formatter/format_elements/bfe_arxiv_link.py deleted file mode 100644 index 8fa7db2eb..000000000 --- a/invenio/modules/formatter/format_elements/bfe_arxiv_link.py +++ /dev/null @@ -1,47 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Links to arXiv""" - -from cgi import escape -from invenio.base.i18n import gettext_set_language - -def format_element(bfo, tag="037__", target="_blank"): - """ - Extracts the arXiv preprint information and - presents it as a direct link towards arXiv.org - """ - _ = gettext_set_language(bfo.lang) - potential_arxiv_ids = bfo.fields(tag) - arxiv_id = "" - for potential_arxiv_id in potential_arxiv_ids: - if potential_arxiv_id.get('9') == 'arXiv' and potential_arxiv_id.get('a', '').startswith('arXiv:'): - arxiv_id = potential_arxiv_id['a'][len('arXiv:'):] - return '%s' % ( - escape(arxiv_id, True), - escape(target, True), - escape(_("This article on arXiv.org"), True), - escape(arxiv_id)) - return "" - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - diff --git a/invenio/modules/formatter/format_elements/bfe_authority_author.py b/invenio/modules/formatter/format_elements/bfe_authority_author.py deleted file mode 100644 index 362fc78d5..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_author.py +++ /dev/null @@ -1,75 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints author data from an Authority Record. -""" -from invenio.config import CFG_SITE_URL, CFG_SITE_NAME -from urllib import quote -import re - -__revision__ = "$Id$" - -def format_element(bfo, main_name='yes', detail='no'): - """ Prints the data of an author authority record in HTML. By default prints - brief version. - - @param detail: whether the 'detailed' rather than the 'brief' format - @type detail: 'yes' or 'no' - """ - - from invenio.base.i18n import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - # return value - out = "" - # local function - def stringify_dict(d): - """ return string composed values in d """ - _str = "" - if 'a' in d: - _str += d['a'] - if 'd' in d: - _str += ", " + d['d'] - return _str or '' - # brief - main_dicts = bfo.fields('100%%') - if len(main_dicts) and main_name=='yes': - main_dict = main_dicts[0] - main = stringify_dict(main_dict) - ## out += "

" + "" + _("Main %s name") % _("author") + "" + ": " + main + "

" - out += "" + main + "" - # detail - if detail.lower() == "yes": - sees = [stringify_dict(see_dict) for see_dict in bfo.fields('400%%')] - sees = filter(None, sees) # fastest way to remove empty ""s - sees = [re.sub(",{2,}",",", x) for x in sees] # prevent ",," - if len(sees): - out += "

" + "" + _("Name variant(s)") + "" + ":

  • " + "
  • ".join(sees) + "

" - see_alsos = [stringify_dict(see_also_dict) for see_also_dict in bfo.fields('500%%')] - see_alsos = filter(None, see_alsos) # fastest way to remove empty ""s - see_alsos = [re.sub(",{2,}",",", x) for x in see_alsos] # prevent ",," - if len(see_alsos): - out += "

" + "" + _("See also") + "" + ":

  • " + ",
  • ".join(see_alsos) + "

" - # return - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_author_publications.py b/invenio/modules/formatter/format_elements/bfe_authority_author_publications.py deleted file mode 100644 index bc2229d1d..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_author_publications.py +++ /dev/null @@ -1,101 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the control number of an Authority Record. -""" - -from invenio.config import CFG_SITE_URL, CFG_SITE_NAME - -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \ - CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD, \ - CFG_BIBAUTHORITY_RECORD_AUTHOR_CONTROL_NUMBER_FIELDS -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no, \ - get_dependent_records_for_control_no - -from invenio.legacy.search_engine import get_fieldvalues - -CFG_BIBAUTHORITY_PUBLICATION_VIEW_LIMIT = 10 -__revision__ = "$Id$" - -def format_element(bfo): - """ Prints the control number of an author authority record in HTML. - By default prints brief version. - - @param brief: whether the 'brief' rather than the 'detailed' format - @type brief: 'yes' or 'no' - """ - - from invenio.messages import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - - - control_nos = [d['a'] for d in bfo.fields('035__') if d['a']] - previous_recIDs = [] - parameters = [] - count = None - publications_formatted = [] - ## for every control number that this author has, find all the connected records for each one - for control_no in control_nos: - for ctrl_number_field_numbers in CFG_BIBAUTHORITY_RECORD_AUTHOR_CONTROL_NUMBER_FIELDS: - parameters.append(ctrl_number_field_numbers + ":" + control_no.replace(" ","")) - recIDs = [x for x in get_dependent_records_for_control_no(control_no) if x not in previous_recIDs] - count = len(recIDs) - count_string = str(count) + " dependent records" - from urllib import quote - # if we have dependent records, provide a link to them - if count: - prefix_pattern = "" - postfix = "" - url_str = '' - # print as many of the author's publications as the CFG_BIBAUTHORITY_PUBLICATION_VIEW_LIMIT allows - for i in range(count if count" - if publications_formatted: - content = "
  • " + "
  • ".join(publications_formatted) + "
" - else: - content = "Missing !" - - p_val = quote(" or ".join(parameters)) - # include "&c=" parameter for bibliographic records - # and one "&c=" parameter for authority records - url_str = \ - "/search" + \ - "?p=" + p_val + \ - "&c=" + quote(CFG_SITE_NAME) + \ - "&c=" + CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME + \ - "&sc=1" + \ - "&ln=" + bfo.lang - prefix = prefix_pattern % url_str - content += prefix + "See all " + str(count) + " publications..." + postfix - - return "

" + title + ": " + content + "

" - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_control_no.py b/invenio/modules/formatter/format_elements/bfe_authority_control_no.py deleted file mode 100644 index 7bd94716c..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_control_no.py +++ /dev/null @@ -1,126 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the control number of an Authority Record. -""" - -from invenio.config import CFG_SITE_URL, CFG_SITE_NAME - -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \ - CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD, \ - CFG_BIBAUTHORITY_RECORD_AUTHOR_CONTROL_NUMBER_FIELDS as control_number_fields, \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_IDENTIFIER as authority_identifier -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no, \ - get_dependent_records_for_control_no - -__revision__ = "$Id$" - -def format_element(bfo): - """ Prints the control number of an author authority record in HTML. - By default prints brief version. - - @param brief: whether the 'brief' rather than the 'detailed' format - @type brief: 'yes' or 'no' - """ - - from invenio.base.i18n import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - - control_nos = [d['a'] for d in bfo.fields('035__') if d.get('a')] - control_nos.extend([d['a'] for d in bfo.fields('970__') if d.get('a')]) - - authority_type = [d.get('a') for d in bfo.fields('980__') if d.get('a') and d.get('a')!=authority_identifier] - if authority_type and type(authority_type) is list: - authority_type = authority_type[0] - - related_control_number_fields = ['510','970'] - related_control_number_fields.extend(control_number_fields.get(authority_type,[])) - control_nos_formatted = [] - for control_no in control_nos: - recIDs = get_dependent_records_for_control_no(control_no) - count = len(recIDs) - count_string = str(count) + " dependent records" - from urllib import quote - # if we have dependent records, provide a link to them - if count: - prefix_pattern = "" - postfix = "" - url_str = '' - # we have multiple dependent records - if count > 1: - # joining control_nos might be more helpful for the user - # than joining recIDs... or maybe not... - parameters = [] - for control_number_field in related_control_number_fields: - parameters.append(control_number_field + ":" + control_no ) - p_val = quote(" or ".join(parameters)) - # include "&c=" parameter for bibliographic records - # and one "&c=" parameter for authority records - url_str = \ - "/search" + \ - "?p=" + p_val + \ - "&c=" + quote(CFG_SITE_NAME) + \ - "&c=" + CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME + \ - "&sc=1" + \ - "&ln=" + bfo.lang - # we have exactly one dependent record - elif count == 1: - url_str = "/record/" + str(recIDs[0]) - - prefix = prefix_pattern % (url_str) - count_string = prefix + count_string + postfix - #assemble the html and append to list - html_str = control_no + " (" + count_string + ")" - - # check if there are more than one authority record with the same - # control number. If so, warn the user about this inconsistency. - # TODO: hide this warning from unauthorized users - my_recIDs = get_low_level_recIDs_from_control_no(control_no) - if len(my_recIDs) > 1: - url_str = \ - "/search" + \ - "?p=" + CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD + ":" + control_no + \ - "&c=" + quote(CFG_SITE_NAME) + \ - "&c=" + CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME + \ - "&sc=1" + \ - "&ln=" + bfo.lang - html_str += \ - ' ' + \ - '(Warning, there is currently ' + \ - 'more than one authority record ' + \ - 'with this Control Number)' + \ - '' - - control_nos_formatted.append(html_str) - - title = "" + _("Control Number(s)") + "" - if control_nos_formatted: - content = "
  • " + "
  • ".join(control_nos_formatted) + "
" - else: - content = "Missing !" - - return "

" + title + ": " + content + "

" - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_institute.py b/invenio/modules/formatter/format_elements/bfe_authority_institute.py deleted file mode 100644 index 47a799b12..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_institute.py +++ /dev/null @@ -1,206 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints institute data from an Authority Record. -""" - -__revision__ = "$Id$" - -from invenio.config import CFG_SITE_URL -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD, \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME - -from invenio.legacy.bibauthority.engine import \ - get_control_nos_from_recID, \ - guess_main_name_from_authority_recID -from invenio.legacy.search_engine import \ - perform_request_search, \ - get_record - -def format_element(bfo, main_name='yes', detail='no'): - """ Prints the data of an institute authority record in HTML. By default prints - brief version. - - @param detail: whether the 'detailed' rather than the 'brief' format - @type detail: 'yes' or 'no' - """ - from invenio.base.i18n import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - - # return value - out = "" - # brief - main_dicts = bfo.fields('110%%') - if len(main_dicts) and main_name=='yes': - main = main_dicts[0].get('a') or "" - ##out += "

" + main + "" - # detail - if detail.lower() == "yes": - sees = [see_dict['a'] for see_dict in bfo.fields('410%%') if 'a' in see_dict] - sees = filter(None, sees) # fastest way to remove empty ""s - if len(sees): - out += "

" + "" + _("Variant(s)") + "" + ": " + ", ".join(sees) + "

" - see_also_dicts = bfo.fields('510%%') - cc_val = CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME - c_val = "Institutes" - record_url_pattern = "/record/" + "%s" - search_url_pattern = "/search?" + \ - "cc=" + "%s" + \ - "&c=" + "%s" + \ - "&p=" + "%s" + \ - "&sc=" + "%s" - link_pattern = "" + '%s' + "" - # populate the first 3 lists - parent_htmls, predecessor_htmls, successor_htmls = \ - get_main_htmls(see_also_dicts, cc_val, c_val, record_url_pattern, - search_url_pattern, link_pattern) - # populate the list of children - child_htmls = \ - get_child_htmls(bfo.recID, cc_val, c_val, record_url_pattern, - link_pattern) - # put it all together - if len(parent_htmls): - out += "

" + "" + _("Parent") + "" + ": " + ", ".join(parent_htmls) + "

" - if len(child_htmls): - out += "

" + "" + _("Children") + "" + ": " + ", ".join(child_htmls) + "

" - if len(predecessor_htmls): - out += "

" + "" + _("Predecessor") + "" + ": " + ", ".join(predecessor_htmls) + "

" - if len(successor_htmls): - out += "

" + "" + _("Successor") + "" + ": " + ", ".join(successor_htmls) + "

" - # return - return out - -def get_main_htmls(see_also_dicts, cc_val, c_val, record_url_pattern, - search_url_pattern, link_pattern): - """parent_htmls, predecessor_htmls, successor_htmls can all be deduced - directly from the metadata of the record""" - # reusable vars - f_val = CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD - sc_val = "1" - parent_htmls = [] - predecessor_htmls = [] - successor_htmls = [] - - # start processing - for see_also_dict in see_also_dicts: - if 'w' in see_also_dict: - # $w contains 'a' for predecessor, 'b' for successor, etc. - w_subfield = see_also_dict.get('w') - # $4 contains control_no of linked authority record - _4_subfield = see_also_dict.get('4') - # $a contains the name of the linked institute - out_string = see_also_dict.get('a') or _4_subfield - # if we have something to display - if out_string: - url = '' - # if we have a control number - if _4_subfield: - p_val = _4_subfield -# if CFG_BIBAUTHORITY_PREFIX_SEP in _4_subfield: -# unused, p_val = _4_subfield.split(CFG_BIBAUTHORITY_PREFIX_SEP); - recIDs = perform_request_search(cc=cc_val, - c=c_val, - p=p_val, - f=f_val) - if len(recIDs) == 1: - url = record_url_pattern % (recIDs[0]) - elif len(recIDs) > 1: - p_val = "recid:" + \ - " or recid:".join([str(r) for r in recIDs]) - url = search_url_pattern % (cc_val, - c_val, - p_val, - sc_val) - # if we found one or multiple records for the control_no, - # make the out_string a clickable url towards those records - if url: - out_string = link_pattern % (url, out_string) - # add the out_string to the appropriate list - if w_subfield == 't': - parent_htmls.append(out_string) - elif w_subfield == 'a': - predecessor_htmls.append(out_string) - elif w_subfield == 'b': - successor_htmls.append(out_string) - # return - return parent_htmls, predecessor_htmls, successor_htmls - -def get_child_htmls(this_recID, cc_val, c_val, record_url_pattern, - link_pattern): - """children aren'r referenced by parents, so we need special treatment to find - them""" - control_nos = get_control_nos_from_recID(this_recID) - for control_no in control_nos: - url = '' - p_val = '510%4:"' + control_no + '" and 510%w:t' - # find a first, fuzzy result set - # narrowing down on a few possible recIDs - recIDs = perform_request_search(cc=cc_val, - c=c_val, - p=p_val) - # now filter to find the ones where the subfield conditions of p_val - # are both true within the exact same field - sf_req = [('w', 't'), ('4', control_no)] - recIDs = filter(lambda x: - match_all_subfields_for_tag(x, '510', sf_req), - recIDs) - # proceed with assembling the html link - child_htmls = [] - for recID in recIDs: - url = record_url_pattern % str(recID) - display = guess_main_name_from_authority_recID(recID) or str(recID) - out_html = link_pattern % (url, display) - child_htmls.append(out_html) - return child_htmls - -def match_all_subfields_for_tag(recID, field_tag, subfields_required=[]): - """ - Tests whether the record with recID has at least one field with 'field_tag' - where all of the required subfields in subfields_required match a subfield - in the given field both in code and value - - @param recID: record ID - @type recID: int - - @param field_tag: a 3 digit code for the field tag code - @type field_tag: string - - @param subfields_required: a list of subfield code/value tuples - @type subfields_required: list of tuples of strings. - same format as in get_record(): - e.g. [('w', 't'), - ('4', 'XYZ123')] - - @return: boolean - """ - rec = get_record(recID) - for field in rec[field_tag]: - subfields_present = field[0] - intersection = set(subfields_present) & set(subfields_required) - if set(subfields_required) == intersection: - return True - return False - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_journal.py b/invenio/modules/formatter/format_elements/bfe_authority_journal.py deleted file mode 100644 index 5fedcde03..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_journal.py +++ /dev/null @@ -1,71 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints journal data from an Authority Record. -""" - -import re - -__revision__ = "$Id$" - -def format_element(bfo, detail='no'): - """ Prints the data of a journal authority record in HTML. By default prints - brief version. - - @param detail: whether the 'detailed' rather than the 'brief' format - @type detail: 'yes' or 'no' - """ - from invenio.base.i18n import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - # return value - out = "" - # local function - def stringify_dict(d): - """ return string composed values in d """ - _str = "" - if 'a' in d: - _str += d['a'] - return _str or '' - # brief - main_dicts = bfo.fields('130%%') - if len(main_dicts): - main_dict = main_dicts[0] - main = stringify_dict(main_dict) - ##out += "

" + "" + _("Main %s name") % _("journal") + "" + ": " + main + "

" - out += "" + main + "" - # detail - if detail.lower() == "yes": - sees = [stringify_dict(see_dict) for see_dict in bfo.fields('430%%')] - sees = filter(None, sees) # fastest way to remove empty ""s - sees = [re.sub(",{2,}",",", x) for x in sees] # prevent ",," - if len(sees): - out += "

" + "" + _("Variant(s)") + "" + ": " + ", ".join(sees) + "

" - see_alsos = [stringify_dict(see_also_dict) for see_also_dict in bfo.fields('530%%')] - see_alsos = filter(None, see_alsos) # fastest way to remove empty ""s - see_alsos = [re.sub(",{2,}",",", x) for x in see_alsos] # prevent ",," - if len(see_alsos): - out += "

" + "" + _("See also") + "" + ": " + ", ".join(see_alsos) + "

" - # return - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_links.py b/invenio/modules/formatter/format_elements/bfe_authority_links.py deleted file mode 100644 index 0c52b5ce2..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_links.py +++ /dev/null @@ -1,95 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the control number of an Authority Record. -""" - -from invenio.config import CFG_SITE_URL, CFG_SITE_NAME - -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \ - CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD, \ - CFG_BIBAUTHORITY_RECORD_AUTHOR_CONTROL_NUMBER_FIELDS -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no, \ - get_dependent_records_for_control_no - -from invenio.utils.viaf import get_wikipedia_link,get_wiki_link_from_record - -__revision__ = "$Id$" - -def format_element(bfo): - """ Prints the control number of an author authority record in HTML. - By default prints brief version. - - @param brief: whether the 'brief' rather than the 'detailed' format - @type brief: 'yes' or 'no' - """ - - from invenio.messages import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - - control_nos = [d['a'] for d in bfo.fields('035__') if d['a'] is not None] - control_nos = filter(None, control_nos) # fastest way to remove empty ""s - - style = "style='width:auto;height:20px;margin-right:10px'" - links_formatted = [] - for control_no in control_nos: - from urllib import quote - image_pattern = "%(text)s" - - if (control_no.find("|(VIAF)") != -1): - viaf_id = control_no.split("|(VIAF)")[1] - link_to_wikipedia = get_wiki_link_from_record(bfo) - if not link_to_wikipedia: - link_to_wikipedia = get_wikipedia_link(viaf_id) - ## Wikipedia link with wiki icon - if link_to_wikipedia: - image_element = image_pattern % { "text": "Wikipedia link", "image": "wikipedia.png", "external_article": link_to_wikipedia} - links_formatted.append(image_element) - ## VIAF link - image_element = image_pattern \ - % { "text" : "VIAF cluster link","image": "viaf.png", "external_article": str("http://viaf.org/viaf/"+viaf_id) } - links_formatted.append(image_element) - ## Library of congress link - if (control_no.find("|(DLC)") != -1): - dlc_id = control_no.split("|(DLC)")[1].replace(" ","") - link_to_lccn = "http://lccn.loc.gov/"+ dlc_id - image_element = image_pattern % { "text": "Library of Congress link", "image": "library_of_congress.png", "external_article" : link_to_lccn } - links_formatted.append(image_element) - - - - - - if links_formatted: - title = "" + _("Useful links") + "" - if links_formatted: - content = "
  • " + "
  • ".join(links_formatted) + "
" - else: - content = "Missing !" - - return "

" + title + ": " + content + "

" - else: - return None -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_publications.py b/invenio/modules/formatter/format_elements/bfe_authority_publications.py deleted file mode 100644 index 42f75e63d..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_publications.py +++ /dev/null @@ -1,118 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the control number of an Authority Record. -""" - -from invenio.config import CFG_SITE_URL, CFG_SITE_NAME - -from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \ - CFG_BIBAUTHORITY_RECORD_CONTROL_NUMBER_FIELD, \ - CFG_BIBAUTHORITY_RECORD_AUTHOR_CONTROL_NUMBER_FIELDS as control_number_fields, \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_IDENTIFIER as authority_identifier -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no, \ - get_dependent_records_for_control_no - -from invenio.legacy.search_engine import get_fieldvalues - -CFG_BIBAUTHORITY_PUBLICATION_VIEW_LIMIT = 10 -__revision__ = "$Id$" - -def format_element(bfo): - """ Prints the control number of an author authority record in HTML. - By default prints brief version. - - @param brief: whether the 'brief' rather than the 'detailed' format - @type brief: 'yes' or 'no' - """ - - from invenio.messages import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - - - control_nos = [d['a'] for d in bfo.fields('035__') if d.get('a')] - authority_type = [d['a'] for d in bfo.fields('980__') if d.get('a') and d.get('a')!=authority_identifier] - if authority_type and type(authority_type) is list: - authority_type = authority_type[0] - - - previous_recIDs = [] - parameters = [] - count = 0 - publications_formatted = [] - recids_added = set() - ## for every control number that this author has, find all the connected records for each one - for control_no in control_nos: - for control_number_field in control_number_fields.get(authority_type,[]): - parameters.append(control_number_field + ":" + control_no.replace(" ","")) - recIDs = [x for x in get_dependent_records_for_control_no(control_no) if x not in previous_recIDs] - length = len(recIDs) or None - from urllib import quote - # if we have dependent records, provide a link to them - if length: - prefix_pattern = "" - postfix = "" - url_str = '' - # print as many of the author's publications as the CFG_BIBAUTHORITY_PUBLICATION_VIEW_LIMIT allows - for i in range(length if length" - content = "" - if publications_formatted: - content = "
  • " + "
  • ".join(publications_formatted) + "
" - #else: - # content = "Missing !" - - p_val = quote(" or ".join(parameters)) - # include "&c=" parameter for bibliographic records - # and one "&c=" parameter for authority records - url_str = \ - "/search" + \ - "?p=" + p_val + \ - "&c=" + quote(CFG_SITE_NAME) + \ - "&c=" + CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME + \ - "&sc=1" + \ - "&ln=" + bfo.lang - prefix = prefix_pattern % url_str - if content: - content += prefix + "See all " + str(count) + " publications..." + postfix - return "

" + title + ": " + content + "

" - else: - return "" - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authority_subject.py b/invenio/modules/formatter/format_elements/bfe_authority_subject.py deleted file mode 100644 index 55ff7a46e..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authority_subject.py +++ /dev/null @@ -1,71 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints subject data from an Authority Record. -""" - -import re - -__revision__ = "$Id$" - -def format_element(bfo, detail='no'): - """ Prints the data of a subject authority record in HTML. By default prints - brief version. - - @param detail: whether the 'detailed' rather than the 'brief' format - @type detail: 'yes' or 'no' - """ - from invenio.base.i18n import gettext_set_language - _ = gettext_set_language(bfo.lang) # load the right message language - # return value - out = "" - # local function - def stringify_dict(d): - """ return string composed values in d """ - _str = "" - if 'a' in d: - _str += d['a'] - return _str or '' - # brief - main_dicts = bfo.fields('150%%') - if len(main_dicts): - main_dict = main_dicts[0] - main = stringify_dict(main_dict) - out += "" + main + "" - ##out += "

" + "" + _("Main %s name") % _("subject") + "" + ": " + main + "

" - # detail - if detail.lower() == "yes": - sees = [stringify_dict(see_dict) for see_dict in bfo.fields('450%%')] - sees = filter(None, sees) # fastest way to remove empty ""s - sees = [re.sub(",{2,}",",", x) for x in sees] # prevent ",," - if len(sees): - out += "

" + "" + _("Variant(s)") + "" + ": " + ", ".join(sees) + "

" - see_alsos = [stringify_dict(see_also_dict) for see_also_dict in bfo.fields('550%%')] - see_alsos = filter(None, see_alsos) # fastest way to remove empty ""s - see_alsos = [re.sub(",{2,}",",", x) for x in see_alsos] # prevent ",," - if len(see_alsos): - out += "

" + "" + _("See also") + "" + ": " + ", ".join(see_alsos) + "

" - # return - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authors.py b/invenio/modules/formatter/format_elements/bfe_authors.py deleted file mode 100644 index 7c4dc5559..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authors.py +++ /dev/null @@ -1,255 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - Print authors.""" - -import re -from urllib import quote -from cgi import escape -from invenio.base.i18n import gettext_set_language -from invenio.base.globals import cfg -from invenio.legacy.bibauthority.engine import \ - get_low_level_recIDs_from_control_no -from invenio.ext.logging import register_exception - - -def format_element(bfo, limit, separator=' ; ', - extension='[...]', - print_links="yes", - print_affiliations='no', - affiliation_prefix=' (', - affiliation_suffix=')', - interactive="no", - highlight="no", - link_author_pages="no", - link_mobile_pages="no", - relator_code_pattern=None, - multiple_affiliations="no", - print_orcid="no", - orcid_type="text", - orcid_text="no", - orcid_prefix="[", - orcid_postfix="]"): - """ - Print the list of authors of a record. - - @param limit: the maximum number of authors to display - @param separator: the separator between authors. - @param extension: a text printed if more authors than 'limit' exist - @param print_links: if yes, prints the authors as HTML link to their publications - @param print_affiliations: if yes, make each author name followed by its affiliation - @param affiliation_prefix: prefix printed before each affiliation - @param affiliation_suffix: suffix printed after each affiliation - @param interactive: if yes, enable user to show/hide authors when there are too many (html + javascript) - @param highlight: highlights authors corresponding to search query if set to 'yes' - @param link_author_pages: should we link to author pages if print_links in on? - @param link_mobile_pages: should we link to mobile app pages if print_links in on? - @param relator_code_pattern: a regular expression to filter authors based on subfield $4 (relator code) - @param multiple_affiliations: whether all affiliations should be displayed - @param print_orcid: if yes, make each author name followed by its ORCID - @param orcid_type: the type of ORCID to be displayed. Accepted values: logo link, text - @param orcid_text: text to put in a link, if left blank it will use an ORCID - @param orcid_prefix: prefix for link and plain text - @param orcid_postfix: postfix for link and plain text - """ - CFG_BASE_URL = cfg['CFG_BASE_URL'].encode('utf-8') - CFG_SITE_RECORD = cfg['CFG_BASE_URL'].encode('utf-8') - - from invenio.legacy.bibauthority.config import \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME, \ - CFG_BIBAUTHORITY_TYPE_NAMES, \ - CFG_BIBAUTHORITY_PREFIX_SEP - - _ = gettext_set_language(bfo.lang) # load the right message language - - authors = [] - authors_1 = bfo.fields('100__', repeatable_subfields_p=True) - authors_2 = bfo.fields('700__', repeatable_subfields_p=True) - - authors.extend(authors_1) - authors.extend(authors_2) - - # make unique string per key - for author in authors: - if 'a' in author: - author['a'] = author['a'][0] - if 'u' in author and multiple_affiliations == 'no': - author['u'] = author['u'][0] - if 'v' in author and multiple_affiliations == 'no': - author['v'] = author['v'][0] - pattern = '%s' + CFG_BIBAUTHORITY_PREFIX_SEP + "(" - for control_no in author.get('0', []): - if pattern % (CFG_BIBAUTHORITY_TYPE_NAMES["INSTITUTE"]) in control_no: - author['u0'] = control_no # overwrite if multiples - elif pattern % (CFG_BIBAUTHORITY_TYPE_NAMES["AUTHOR"]) in control_no: - author['a0'] = control_no # overwrite if multiples - - if relator_code_pattern: - p = re.compile(relator_code_pattern) - authors = filter(lambda x: p.match(x.get('4', '')), authors) - - nb_authors = len(authors) - - bibrec_id = bfo.control_field("001") - - # Process authors to add orcid, link, highlight and format affiliation - for author in authors: - - if 'a' in author: - if highlight == 'yes': - from invenio.modules.formatter import utils as bibformat_utils - author['a'] = bibformat_utils.highlight(author['a'], - bfo.search_pattern) - - if print_links.lower() == "yes": - if link_author_pages == "yes": - author['a'] = '' - elif link_mobile_pages == 'yes': - author['a'] = '' + escape(author['a']) + '' - else: - auth_coll_param = '' - if 'a0' in author: - recIDs = get_low_level_recIDs_from_control_no( - author['a0']) - if len(recIDs): - auth_coll_param = '&c=' + \ - CFG_BIBAUTHORITY_AUTHORITY_COLLECTION_NAME - author['a'] = '' + escape(author['a']) + '' - else: - author['a'] = '' + escape(author['a']) + '' - - if 'u' in author or 'v' in author: - if print_affiliations == "yes": - if 'u0' in author: - recIDs = get_low_level_recIDs_from_control_no(author['u0']) - # if there is more than 1 recID, clicking on link and - # thus displaying the authority record's page should - # contain a warning that there are multiple authority - # records with the same control number - if isinstance(author['u'], (list, tuple)): - author['u'] = author['u'][0] - if len(recIDs): - author['u'] = '' + author['u'] + '' - if 'u' not in author and 'v' in author: - author['u'] = author['v'] - if isinstance(author['u'], (list, tuple)): - author['u'] = ' '.join([affiliation_prefix + aff + - affiliation_suffix for aff in author['u']]) - else: - author['u'] = affiliation_prefix + author['u'] + \ - affiliation_suffix - - if 'j' in author: - if print_orcid == "yes": - orcid = author.get('j', "") - if orcid[0]: - orcid = orcid[0].split(':')[1] - if orcid_type == 'logo': - author['j'] = ' ' % (orcid, orcid) - elif orcid_type == 'link': - if orcid_text == "no": - author['j'] = '%s%s%s' % (orcid_prefix, orcid, orcid, orcid_postfix) - else: - author['j'] = '%s%s%s' % (orcid_prefix, orcid, orcid_text, orcid_postfix) - else: - author['j'] = '%s%s%s' % (orcid_prefix, orcid, orcid_postfix) - else: - author['j'] = "" - - # Flatten author instances - new_authors = [] - for author in authors: - auth = author.get('a', '') - if print_orcid == 'yes': - auth = auth + author.get('j', '') - if print_affiliations == 'yes': - auth = auth + author.get('u', '') - new_authors.append(auth) - authors = new_authors - - if limit.isdigit() and nb_authors > int(limit) and interactive != "yes": - return separator.join(authors[:int(limit)]) + extension - - elif limit.isdigit() and nb_authors > int(limit) and interactive == "yes": - out = '' - out += separator.join(authors[:int(limit)]) - out += '' % bibrec_id + separator + \ - separator.join(authors[int(limit):]) + '' - out += ' ' % bibrec_id - out += ' ' % bibrec_id - out += ''' - - ''' % {'show_less': _("Hide"), - 'show_more': _("Show all %(x_num)i authors", x_num=nb_authors), - 'extension': extension, - 'recid': bibrec_id} - out += '' % bibrec_id - - return out - elif nb_authors > 0: - return separator.join(authors) - - -def escape_values(bfo): - """Escape values. - - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_authors_parent.py b/invenio/modules/formatter/format_elements/bfe_authors_parent.py deleted file mode 100644 index dae1b9670..000000000 --- a/invenio/modules/formatter/format_elements/bfe_authors_parent.py +++ /dev/null @@ -1,359 +0,0 @@ -# -*- coding: utf-8 -*- -# -# $Id$ -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints authors -""" -__revision__ = "$Id$" - - -def format_element(bfo, limit, separator='; ', - extension='[...]', - print_links="yes", - print_affiliations='no', - affiliation_prefix=' (', - affiliation_suffix=')', - print_affiliation_first='no', - interactive="no", - highlight="no", - affiliations_separator=" ; ", - name_last_first="yes", - collaboration="yes", - id_links="no", - markup="html", - link_extension="no", - suffix='' - ): - """ - Prints the list of authors of a record. - - @param limit the maximum number of authors to display - @param separator the separator between authors. - @param extension a text printed if more authors than 'limit' exist - @param print_links if yes, prints the authors as HTML link to their publications - @param print_affiliations if yes, make each author name followed by its affiliation - @param affiliation_prefix prefix printed before each affiliation - @param affiliation_suffix suffix printed after each affiliation - @param print_affiliation_first if 'yes', affiliation is printed before the author - @param interactive if yes, enable user to show/hide authors when there are too many (html + javascript) - @param highlight highlights authors corresponding to search query if set to 'yes' - @param affiliations_separator separates affiliation groups - @param name_last_first if yes (default) print last, first otherwise first last - @param collaboration if yes (default) uses collaboration name in place of long author list, if available - @param id_links if yes (default = no) prints link based on INSPIRE IDs if available - only used if print_links = yes - @param markup html (default) or latex controls small markup differences - @param link_extension if 'yes' link the extension to the detailed - record page - - """ - from urllib import quote - from cgi import escape - import re - from invenio.base.i18n import gettext_set_language - from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD - from invenio.modules.formatter.engine import BibFormatObject - - _ = gettext_set_language(bfo.lang) # load the right message language - - #regex for parsing last and first names and initials - re_last_first = re.compile('^(?P[^,]+)\s*,\s*(?P[^\,]*)(?P\,?.*)$') - re_initials = re.compile(r'(?P\w)(\w+|\.)\s*') - re_coll = re.compile(r'\s*collaborations?', re.IGNORECASE) - - bibrec_id = bfo.control_field("001") - authors = [] - lastauthor = '' - - # HepData and only-INSPIRE data records inherit the list of authors from the original paper - if (bfo.field("520__9") == "HEPDATA") or (bfo.field("520__9") == "INSPIRE"): - parent_recid = bfo.field("786__w") - bfo_parent = BibFormatObject(int(parent_recid)) - - authors = [] - authors_1 = bfo_parent.fields('100__', repeatable_subfields_p=True) - authors_2 = bfo_parent.fields('700__', repeatable_subfields_p=True) - # other datasources should have a list of authors - else: - authors = [] - authors_1 = bfo.fields('100__', repeatable_subfields_p=True) - authors_2 = bfo.fields('700__', repeatable_subfields_p=True) - - authors.extend(authors_1) - authors.extend(authors_2) - - # If there are no author check for corporate author in 110__a field - if len(authors) == 0: - authors = bfo.fields('110__', repeatable_subfields_p=True) - # For corporate authors we don't want to reverse names order - name_last_first = 'yes' - # And we don't want to create links - print_links = 'no' - - # Keep real num of authors. fix + affiliations_separator.join(author['u']) + \ - nb_authors = len(authors) - - # Limit num of authors, so that we do not process - # the authors that will not be shown. This can only - # be done in non-interactive mode, as interactive mode - # allows to show all of them. - if limit.isdigit() and nb_authors > int(limit) \ - and interactive != "yes": - if bfo.field('710g'): # check for colln note - authors = authors[:1] - else: - - authors = authors[:int(limit)] - - # Process authors to add link, affiliation and highlight - for author in authors: - - if author.has_key('a'): - author['a'] = author['a'][0] # There should not be - # repeatable subfields here. - if highlight == 'yes': - from invenio import bibformat_utils - author['a'] = bibformat_utils.highlight(author['a'], - bfo.search_pattern) - - #check if we need to reverse last, first - #we don't try to reverse it if it isn't stored with a comma. - first_last_match = re_last_first.search(author['a']) - author['display'] = author['a'] - - if name_last_first.lower() == "no": - if first_last_match: - author['display'] = first_last_match.group('first_names') + \ - ' ' + \ - first_last_match.group('last') + \ - first_last_match.group('extension') - - #for latex we do initials only (asn assume first last) - if markup == 'latex': - if first_last_match: - first = re_initials.sub('\g.~', \ - first_last_match.group('first_names')) - author['display'] = first + \ - first_last_match.group('last') + \ - first_last_match.group('extension') - - - if print_links.lower() == "yes": - - # if there is an ID, search using that. - id_link = '' - if id_links == "yes" and author.has_key('i'): - author['i'] = author['i'][0] #possible to have more IDs? - id_link = ''+escape("(ID Search)") + ' ' - - - author['display'] = '' + escape(author['display'])+'' + \ - id_link - - if print_affiliations == "yes": - if author.has_key('e'): - author['e'] = affiliation_prefix + \ - affiliations_separator.join(author['e']) + \ - affiliation_suffix - - - - if author.has_key('u'): - author['ilink'] = ['' + \ - string.lstrip() + \ - '' for string in author['u']] - author['u'] = affiliation_prefix + \ - affiliations_separator.join(author['ilink']) + \ - affiliation_suffix - -# -# Consolidate repeated affiliations -# - last = '' - authors.reverse() - for author in authors: - if not author.has_key('u'): - author['u'] = '' - #print 'this->'+ author['a']+'\n' - if last == author['u']: - author['u'] = '' - else: - last = author['u'] - - authors.reverse() - - # Flatten author instances - if print_affiliations == 'yes': -# 100__a (100__e) 700__a (100__e) (100__u) - if print_affiliation_first.lower() != 'yes': - authors = [author.get('display', '') + author.get('e', '') + author.get('u', '') - for author in authors] - - else: - authors = [author.get('u', '') + author.get('display', '') - for author in authors] - - else: - authors = [author.get('display', '') - for author in authors] - - # link the extension to detailed record - if link_extension == 'yes' and interactive != 'yes': - extension = '' + \ - extension + '' - - # Detect Collaborations: - if collaboration == "yes": - colls = [] - for coll in bfo.fields("710__g"): - if coll not in colls: - colls.append(coll) - else: - colls = [] - if colls: - short_coll = False - colls = [re_coll.sub('', coll) for coll in colls] - if print_links.lower() == "yes": - colls = [''+escape(coll)+'' for coll in colls] - - coll_display = " and ".join(colls) - if not coll_display.endswith("aboration"): - coll_display += " Collaboration" - if len(colls) > 1: - coll_display += 's' - if nb_authors > 1: - if markup == 'latex': - coll_display = authors[0] + extension + " [" + \ - coll_display + "]" - elif interactive == "yes": - coll_display += " (" + authors[0] + " " - extension += ")" - else: #html - coll_display += " (" + authors[0] + extension + ")" - elif nb_authors == 1: - short_coll = True - if markup == 'latex': - coll_display = authors[0] + " [" + coll_display + "]" - else: #html - coll_display += " (" + authors[0] + " for the collaboration)" - elif nb_authors == 0: - short_coll = True - if markup == 'latex': - coll_display = "[" + coll_display + "]" - - # Start outputting, depending on options and number of authors - if colls and (interactive != "yes" or short_coll): - return coll_display - - if limit.isdigit() and nb_authors > int(limit) and interactive != "yes": - if markup == 'latex': - lastauthor = authors.pop() - lastauthor = ' and ' + lastauthor - limit = int(limit) - 1 - - return separator.join(authors[:int(limit)]) + lastauthor + \ - extension - - elif interactive == "yes" and ((colls and not short_coll) or (limit.isdigit() and nb_authors > int(limit))): - out = ''' - - ''' % {'show_less': _("Hide"), - 'show_more': _("Show all %(x_num_of_authors)i authors", x_num_of_authors=nb_authors), - 'extension': extension} - -# out += '' - if colls: - show = coll_display - more = separator + separator.join(authors[1:]) + ')' - else: - show = separator.join(authors[:int(limit)]) - more = separator.join(authors[int(limit):len(authors)]) - - out += show - out += ' ' + more + '' - out += ' ' - out += ' ' - out += '' - return out - elif nb_authors > 0: - if markup == 'latex' and nb_authors > 1: - lastauthor = authors.pop() - lastauthor = ' and ' + lastauthor - output = separator.join(authors) + lastauthor - # remove the dot from the end of authors list when the suffix starts with dot - # (to avoid two consecutive dots) - if suffix and output and output[-1] == suffix[0] == '.': - output = output[:-1] - return output - -# we know the argument is unused, thanks -# pylint: disable-msg=W0613 - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 -# pylint: enable-msg=W0613 diff --git a/invenio/modules/formatter/format_elements/bfe_bibtex.py b/invenio/modules/formatter/format_elements/bfe_bibtex.py deleted file mode 100644 index a3be7ff4d..000000000 --- a/invenio/modules/formatter/format_elements/bfe_bibtex.py +++ /dev/null @@ -1,519 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints BibTeX meta-data -""" -__revision__ = "$Id$" - -from invenio.config import CFG_SITE_LANG - -def format_element(bfo, width="50"): - """ - Prints a full BibTeX record. - - 'width' must be bigger than or equal to 30. - This format element is an example of large element, which does - all the formatting by itself - - @param width: the width (in number of characters) of the record - """ - out = "@" - width = int(width) - if width < 30: - width = 30 - - name_width = 20 - value_width = width-name_width - recID = bfo.control_field('001') - - #Print entry type - import invenio.modules.formatter.format_elements.bfe_collection as bfe_collection - collection = bfe_collection.format_element(bfo=bfo, kb="DBCOLLID2BIBTEX") - if collection == "": - out += "article" - else: - out += collection - - out += "{" - - #Print BibTeX key - # - #Try to have: author_name:recID - #If author_name cannot be found, use primary_report_number - #If primary_report_number cannot be found, use additional_report_number - #If additional_report_number cannot be found, use title:recID - #If title cannot be found, use only recID - # - #The construction of this key is inherited from old BibTeX format - #written in EL, in old BibFormat. - key = recID - author = bfo.field("100__a") - if author != "": - key = get_name(author)+":"+recID - else: - author = bfo.field("700__a") - if author != "": - key = get_name(author)+":"+recID - else: - primary_report_number = bfo.field("037__a") - if primary_report_number != "": - key = primary_report_number - else: - additional_report_number = bfo.field("088__a") - if additional_report_number != "": - key = primary_report_number - else: - title = bfo.field("245__a") - if title != "": - key = get_name(title)+":"+recID - out += key +"," - - #Print authors - #If author cannot be found, print a field key=recID - import invenio.modules.formatter.format_elements.bfe_authors as bfe_authors - authors = bfe_authors.format_element(bfo=bfo, - limit="", - separator=" and ", - extension="", - print_links="no") - if authors == "": - out += format_bibtex_field("key", - recID, - name_width, - value_width) - else: - out += format_bibtex_field("author", - authors, - name_width, - value_width) - - #Print editors - import invenio.modules.formatter.format_elements.bfe_editors as bfe_editors - editors = bfe_editors.format_element(bfo=bfo, limit="", - separator=" and ", - extension="", - print_links="no") - out += format_bibtex_field("editor", - editors, - name_width, - value_width) - - #Print title - import invenio.modules.formatter.format_elements.bfe_title as bfe_title - title = bfe_title.format_element(bfo=bfo, separator = ". ") - out += format_bibtex_field("title", - '{' + title + '}', - name_width, - value_width) - - #Print institution - if collection == "techreport": - publication_name = bfo.field("269__b") - out += format_bibtex_field("institution", - publication_name, - name_width, value_width) - - #Print organization - if collection == "inproceedings" or collection == "proceedings": - organization = [] - organization_1 = bfo.field("260__b") - if organization_1 != "": - organization.append(organization_1) - organization_2 = bfo.field("269__b") - if organization_2 != "": - organization.append(organization_2) - out += format_bibtex_field("organization", - ". ".join(organization), - name_width, - value_width) - - #Print publisher - if collection == "book" or \ - collection == "inproceedings" \ - or collection == "proceedings": - publishers = [] - import invenio.modules.formatter.format_elements.bfe_publisher as bfe_publisher - publisher = bfe_publisher.format_element(bfo=bfo) - if publisher != "": - publishers.append(publisher) - publication_name = bfo.field("269__b") - if publication_name != "": - publishers.append(publication_name) - imprint_publisher_name = bfo.field("933__b") - if imprint_publisher_name != "": - publishers.append(imprint_publisher_name) - imprint_e_journal__publisher_name = bfo.field("934__b") - if imprint_e_journal__publisher_name != "": - publishers.append(imprint_e_journal__publisher_name) - - out += format_bibtex_field("publisher", - ". ".join(publishers), - name_width, - value_width) - - #Print journal - if collection == "article": - journals = [] - host_title = bfo.field("773__p") - if host_title != "": - journals.append(host_title) - journal = bfo.field("909C4p") - if journal != "": - journals.append(journal) - - out += format_bibtex_field("journal", - ". ".join(journals), - name_width, - value_width) - - #Print school - if collection == "phdthesis": - university = bfo.field("502__b") - - out += format_bibtex_field("school", - university, - name_width, - value_width) - - # Collaboration - collaborations = [] - for collaboration in bfo.fields("710__g"): - if collaboration not in collaborations: - collaborations.append(collaboration) - out += format_bibtex_field("collaboration", - ", ".join(collaborations), - name_width, - value_width) - - #Print address - if collection == "book" or \ - collection == "inproceedings" or \ - collection == "proceedings" or \ - collection == "phdthesis" or \ - collection == "techreport": - addresses = [] - publication_place = bfo.field("260__a") - if publication_place != "": - addresses.append(publication_place) - publication_place_2 = bfo.field("269__a") - if publication_place_2 != "": - addresses.append(publication_place_2) - imprint_publisher_place = bfo.field("933__a") - if imprint_publisher_place != "": - addresses.append(imprint_publisher_place) - imprint_e_journal__publisher_place = bfo.field("934__a") - if imprint_e_journal__publisher_place != "": - addresses.append(imprint_e_journal__publisher_place) - - out += format_bibtex_field("address", - ". ".join(addresses), - name_width, - value_width) - - #Print number - if collection == "techreport" or \ - collection == "article": - numbers = [] - primary_report_number = bfo.field("037__a") - if primary_report_number != "": - numbers.append(primary_report_number) - additional_report_numbers = bfo.fields("088__a") - additional_report_numbers = ". ".join(additional_report_numbers) - if additional_report_numbers != "": - numbers.append(additional_report_numbers) - host_number = bfo.field("773__n") - if host_number != "": - numbers.append(host_number) - number = bfo.field("909C4n") - if number != "": - numbers.append(number) - out += format_bibtex_field("number", - ". ".join(numbers), - name_width, - value_width) - - #Print volume - if collection == "article" or \ - collection == "book": - volumes = [] - host_volume = bfo.field("773__v") - if host_volume != "": - volumes.append(host_volume) - volume = bfo.field("909C4v") - if volume != "": - volumes.append(volume) - - out += format_bibtex_field("volume", - ". ".join(volumes), - name_width, - value_width) - - #Print series - if collection == "book": - series = bfo.field("490__a") - out += format_bibtex_field("series", - series, - name_width, - value_width) - - #Print pages - if collection == "article" or \ - collection == "inproceedings": - pages = [] - host_pages = bfo.field("773c") - if host_pages != "": - pages.append(host_pages) - nb_pages = bfo.field("909C4c") - if nb_pages != "": - pages.append(nb_pages) - phys_pagination = bfo.field("300__a") - if phys_pagination != "": - pages.append(phys_pagination) - - out += format_bibtex_field("pages", - ". ".join(pages), - name_width, - value_width) - - #Print month - month = get_month(bfo.field("269__c")) - if month == "": - month = get_month(bfo.field("260__c")) - if month == "": - month = get_month(bfo.field("502__c")) - - out += format_bibtex_field("month", - month, - name_width, - value_width) - - #Print year - year = get_year(bfo.field("269__c")) - if year == "": - year = get_year(bfo.field("260__c")) - if year == "": - year = get_year(bfo.field("502__c")) - if year == "": - year = get_year(bfo.field("909C0y")) - - out += format_bibtex_field("year", - year, - name_width, - value_width) - - #Print note - note = bfo.field("500__a") - out += format_bibtex_field("note", - note, - name_width, - value_width) - - #Print doi - fields = bfo.fields("0247_") - for field in fields: - if field.get('2', 'DOI') == 'DOI' and 'a' in field: - out += format_bibtex_field("doi", - "{%s}" % field['a'], - name_width, - value_width) - out += format_bibtex_field("url", - "{http://dx.doi.org/%s}" % field['a'], - name_width, - value_width) - - out += "\n}" - - return out - - -def format_bibtex_field(name, value, name_width=20, value_width=40): - """ - Formats a name and value to display as BibTeX field. - - 'name_width' is the width of the name of the field (everything before " = " on first line) - 'value_width' is the width of everything after " = ". - - 6 empty chars are printed before the name, then the name and then it is filled with spaces to meet - the required width. Therefore name_width must be > 6 + len(name) - - Then " = " is printed (notice spaces). - - So the total width will be:: - name_width + value_width + len(" = ") - (3) - - if value is empty string, then return empty string. - - For example format_bibtex_field('author', 'a long value for this record', 13, 15) will - return : - >> - >> name = "a long value - >> for this record", - """ - if name_width < 6 + len(name): - name_width = 6 + len(name) - if value_width < 2: - value_width = 2 - if value is None or value == "": - return "" - - #format name - name = "\n "+name - name = name.ljust(name_width) - - #format value - value = '"'+value+'"' #Add quotes to value - value_lines = [] - last_cut = 0 - cursor = value_width -1 #First line is smaller because of quote - increase = False - while cursor < len(value): - if cursor == last_cut: #Case where word is bigger than the max - #number of chars per line - increase = True - cursor = last_cut+value_width-1 - - if value[cursor] != " " and not increase: - cursor -= 1 - elif value[cursor] != " " and increase: - cursor += 1 - else: - value_lines.append(value[last_cut:cursor]) - last_cut = cursor - cursor += value_width - increase = False - #Take rest of string - last_line = value[last_cut:] - if last_line != "": - value_lines.append(last_line) - - tabs = "".ljust(name_width + 2) - value = ("\n"+tabs).join(value_lines) - - return name + ' = ' + value + "," - -def get_name(string): - """ - Tries to return the last name contained in a string. - - In fact returns the text before any comma in 'string', whith - spaces removed. If comma not found, get longest word in 'string' - - Behaviour inherited from old GET_NAME function defined as UFD in - old BibFormat. We need to return the same value, to keep back - compatibility with already generated BibTeX records. - - Eg: get_name("سtlund, عvind B") returns "سtlund". - """ - names = string.split(',') - - if len(names) == 1: - #Comma not found. - #Split around any space - longest_name = "" - words = string.split() - for word in words: - if len(word) > len(longest_name): - longest_name = word - return longest_name - else: - return names[0].replace(" ", "") - - -def get_year(date, default=""): - """ - Returns the year from a textual date retrieved from a record - - The returned value is a 4 digits string. - If year cannot be found, returns 'default' - Returns first value found. - - @param date: the textual date to retrieve the year from - @param default: a default value to return if year not fount - """ - import re - year_pattern = re.compile(r'\d\d\d\d') - result = year_pattern.search(date) - if result is not None: - return result.group() - - return default - -def get_month(date, ln=CFG_SITE_LANG, default=""): - """ - Returns the year from a textual date retrieved from a record - - The returned value is the 3 letters short month name in language 'ln' - If year cannot be found, returns 'default' - - @param date: the textual date to retrieve the year from - @param default: a default value to return if year not fount - """ - import re - from invenio.utils.date import get_i18n_month_name - from invenio.base.i18n import language_list_long - - #Look for textual month like "Jan" or "sep" or "November" or "novem" - #Limit to CFG_SITE_LANG as language first (most probable date) - #Look for short months. Also matches for long months - short_months = [get_i18n_month_name(month).lower() - for month in range(1, 13)] # ["jan","feb","mar",...] - short_months_pattern = re.compile(r'('+r'|'.join(short_months)+r')', - re.IGNORECASE) # (jan|feb|mar|...) - result = short_months_pattern.search(date) - if result is not None: - try: - month_nb = short_months.index(result.group().lower()) + 1 - return get_i18n_month_name(month_nb, "short", ln) - except: - pass - - #Look for month specified as number in the form 2004/03/08 or 17 02 2004 - #(always take second group of 2 or 1 digits separated by spaces or - etc.) - month_pattern = re.compile(r'\d([\s]|[-/.,])+(?P(\d){1,2})([\s]|[-/.,])') - result = month_pattern.search(date) - if result is not None: - try: - month_nb = int(result.group("month")) - return get_i18n_month_name(month_nb, "short", ln) - except: - pass - - #Look for textual month like "Jan" or "sep" or "November" or "novem" - #Look for the month in each language - - #Retrieve ['en', 'fr', 'de', ...] - language_list_short = [x[0] - for x in language_list_long()] - for lang in language_list_short: #For each language - #Look for short months. Also matches for long months - short_months = [get_i18n_month_name(month, "short", lang).lower() - for month in range(1, 13)] # ["jan","feb","mar",...] - short_months_pattern = re.compile(r'('+r'|'.join(short_months)+r')', - re.IGNORECASE) # (jan|feb|mar|...) - result = short_months_pattern.search(date) - if result is not None: - try: - month_nb = short_months.index(result.group().lower()) + 1 - return get_i18n_month_name(month_nb, "short", ln) - except: - pass - - return default - - diff --git a/invenio/modules/formatter/format_elements/bfe_bookmark.py b/invenio/modules/formatter/format_elements/bfe_bookmark.py deleted file mode 100644 index 22d3fb19e..000000000 --- a/invenio/modules/formatter/format_elements/bfe_bookmark.py +++ /dev/null @@ -1,139 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - bookmark toolbar. - -See . -""" - -from invenio.config import CFG_SITE_URL, CFG_BASE_URL, CFG_SITE_RECORD, CFG_CERN_SITE -from invenio.legacy.search_engine import get_all_restricted_recids -from invenio.utils.html import escape_javascript_string -from invenio.modules.formatter.format_elements.bfe_sciencewise import create_sciencewise_url, \ - get_arxiv_reportnumber -from invenio.legacy.webjournal.utils import \ - parse_url_string, \ - make_journal_url, \ - get_journals_ids_and_names - - -def format_element( - bfo, only_public_records=1, - sites="linkedin,twitter,facebook,google,delicious,sciencewise"): - """Return a snippet of JavaScript needed for displaying a bookmark toolbar. - - :param only_public_records: if set to 1 (the default), prints the box only - if the record is public (i.e. if it belongs to the root colletion and - is accessible to the world). - - :param sites: which sites to enable (default is - 'linkedin,twitter,facebook,google,delicious,sciencewise'). - This should be a comma separated list of strings. - Valid values are available on: - - Note that 'sciencewise' is an ad-hoc service that will be displayed - only in case the record has an arXiv reportnumber and will always - be displayed last. - """ - if int(only_public_records) and \ - bfo.recID not in get_all_restricted_recids(): - return "" - - sitelist = sites.split(',') - sitelist = [site.strip().lower() for site in sitelist] - - sciencewise = False - if 'sciencewise' in sitelist: - sciencewise = True - sitelist.remove('sciencewise') - - sites_js = ", ".join("'%s'" % site for site in sitelist) - - title = bfo.field('245__a') - description = bfo.field('520__a') - - sciencewise_script = "" - if sciencewise: - reportnumber = get_arxiv_reportnumber(bfo) - sciencewise_url = "" - if reportnumber: - sciencewise_url = create_sciencewise_url(reportnumber) - if not sciencewise_url and CFG_CERN_SITE: - sciencewise_url = create_sciencewise_url(bfo.recID, cds=True) - if sciencewise_url: - sciencewise_script = """\ -$.bookmark.addSite('sciencewise', 'ScienceWise.info', '%(siteurl)s/img/sciencewise.png', 'en', 'bookmark', '%(url)s'); -$('#bookmark_sciencewise').bookmark({sites: ['sciencewise']}); -""" % { - 'siteurl': CFG_SITE_URL, - 'url': sciencewise_url.replace("'", r"\'"), - } - - url = '%(siteurl)s/%(record)s/%(recid)s' % \ - {'recid': bfo.recID, - 'record': CFG_SITE_RECORD, - 'siteurl': CFG_BASE_URL} - - args = parse_url_string(bfo.user_info['uri']) - journal_name = args["journal_name"] - if journal_name and \ - (journal_name in [info.get('journal_name', '') for info in get_journals_ids_and_names()]): - # We are displaying a WebJournal article: URL is slightly different - url = make_journal_url(bfo.user_info['uri']) - - return """\ - -
- - - - - -""" % { - 'siteurl': CFG_BASE_URL, - 'sciencewise': sciencewise_script, - 'title': escape_javascript_string(title, - escape_for_html=False, - escape_CDATA=True), - 'description': escape_javascript_string(description, - escape_for_html=False, - escape_CDATA=True), - 'sites_js': sites_js, - 'url': url, - } - - -def escape_values(bfo): - """Check if output of this element should be escaped.""" - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_citation_suggestion_hepdata.py b/invenio/modules/formatter/format_elements/bfe_citation_suggestion_hepdata.py deleted file mode 100644 index 0f416b98e..000000000 --- a/invenio/modules/formatter/format_elements/bfe_citation_suggestion_hepdata.py +++ /dev/null @@ -1,199 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the citation suggestion -""" -__revision__ = "$Id$" - -import re - - -def format_element(bfo): - """ - Prints the citation suggestion. Only for Hepdata and Dataverse by now. - Languages/repos will come. - """ - - publisher = bfo.field("520__9") - if publisher == "HEPDATA": - out = hepdata_cite_as(bfo) - elif publisher == "Dataverse": - out = dataverse_cite_as(bfo) - elif publisher == "INSPIRE": - out = inspire_cite_as(bfo) - - return out - -def hepdata_cite_as(bfo): - """ - HepData format example: - Cite as: The ATLAS Collaboration (2013) HepData, doi: 10.1234/123456 - """ - - from invenio.modules.formatter.engine import BibFormatObject - - colls = [] - for coll in bfo.fields("710__g"): - if coll not in colls: - colls.append(coll) - - parent_recid = bfo.field("786__w") - bfo_parent = BibFormatObject(parent_recid) - year = get_year(bfo_parent) - if year == None: - year = 0 - - publisher = bfo.field("520__9") - if publisher == 'HEPDATA': - publisher = 'HepData' - - pid_type = bfo.field("0247_2") - pid = bfo.field("0247_a") - - out = '' - out += ("Cite as: ") - out += str(colls[0]) - out += ' ( '+ str(year) +' ) ' - out += publisher + ', ' - - if pid_type == 'DOI': - out += ' http://doi.org/' + pid + '' - elif pid_type == 'HDL': - out += ' http://hdl.handle.net/' + pid + '' - elif pid_type == '': - out += '[no persistent identifier assigned]' - - return out - -def dataverse_cite_as(bfo): - """ - Dataverse format example: - Cranmer, Kyle; Allanach, Ben; Lester, Christopher; Weber, Arne, "Replication data for: - "Natural Priors, CMSSM Fits and LHC Weather Forecasts"", http://hdl.handle.net/1902.1/21804 - """ - authors = "" - for auth in bfo.fields("100__a"): - authors += str(auth) + "; " - out = '' - out += ("Cite as: ") - out += authors[:-2] + ", " - - title = bfo.field("245__a") - out += '"' + title + '",
' - - pid_type = bfo.field("0247_2") - pid = bfo.field("0247_a") - - if pid_type == 'HDL': - out += ' http://hdl.handle.net/' + pid + '' - elif pid_type == '': - out += '[no persistent identifier assigned]' - - return out - -def inspire_cite_as(bfo): - """ - INSPIRE format example: - Cite as: The ATLAS Collaboration (2013) INSPIRE, doi: 10.1234/123456 - """ - from invenio.modules.formatter.engine import BibFormatObject - - colls = [] - for coll in bfo.fields("710__g"): - if coll not in colls: - colls.append(coll) - - parent_recid = bfo.field("786__w") - bfo_parent = BibFormatObject(parent_recid) - year = get_year(bfo_parent) - if year == None: - year = 0 - - publisher = bfo.field("520__9") - if publisher == "INSPIRE": - publisher == "INSPIRE-HEP" - - pid_type = bfo.field("0247_2") - pid = bfo.field("0247_a") - - out = '' - out += ("Cite as: ") - out += str(colls[0]) - out += ' ( '+ str(year) +' ) ' - out += publisher + ', ' - - if pid_type == 'DOI': - out += ' http://doi.org/' + pid + '' - elif pid_type == 'HDL': - out += ' http://hdl.handle.net/' + pid + '' - elif pid_type == '': - out += '[no persistent identifier assigned]' - - return out - -def get_year(bfo): - """ - Returns a year of publication for the best available date. Returns None if none found. - - @param bfo: BibFormatObject for current record - @type nfo: object - - @return: integer if the year is found, otherwise None. - """ - from invenio.modules.formatter.format_elements.bfe_INSPIRE_arxiv import get_arxiv - - #true date - date = bfo.field('269__c') - if date: - return date[:4] - - #arxiv date - arxiv = get_arxiv(bfo,category="no") - if arxiv: - date = re.search('(\d+)',arxiv[0]).groups()[0] - if len(date) >=4: - year = date[0:2] - if year > '90': - year='19'+year - else: - year='20'+year - return year - - #journal year - date = bfo.field('773__y') - if date: - return date[:4] - - #date added - date = bfo.field('961__x') - if date: - return date[:4] - - #book year - date = bfo.field('260__c') - if date: - return date[:4] - - return None - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_cited_by.py b/invenio/modules/formatter/format_elements/bfe_cited_by.py deleted file mode 100644 index 4d098afff..000000000 --- a/invenio/modules/formatter/format_elements/bfe_cited_by.py +++ /dev/null @@ -1,59 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints reference to documents citing this one -""" -__revision__ = "$Id$" - -import cgi - -def format_element(bfo, separator='; '): - """ - Prints a list of records citing this record - - @param separator: a separator between citations - """ - from urllib import quote - from invenio.config import CFG_BASE_URL - - primary_report_numbers = bfo.fields('037__a') - additional_report_numbers = bfo.fields('088__a') - - primary_citations = ['' + \ - cgi.escape(report_number) + '' \ - for report_number in primary_report_numbers] - - additional_citations = ['' + \ - cgi.escape(report_number) + '' \ - for report_number in additional_report_numbers] - - citations = primary_citations - citations.extend(additional_citations) - - return separator.join(citations) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_client_info.py b/invenio/modules/formatter/format_elements/bfe_client_info.py deleted file mode 100644 index 875bcf740..000000000 --- a/invenio/modules/formatter/format_elements/bfe_client_info.py +++ /dev/null @@ -1,53 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints client info -""" -__revision__ = "$Id$" - -def format_element(bfo, var=''): - ''' - Print several client specific variables. - @param var: the name of the desired variable. Can be one of: ln, search_pattern, uid, referer, uri, nickname, email - ln: the current language of the user - search_pattern: the list of keywords used by the user - uid: the current user id - referer: the url the user came from - uri: the current uri - nickname: the user nickname - email: the user email - ''' - - if var == '': - out = '' - elif var == 'ln': - out = bfo.lang - elif var == 'search_pattern': - out = ' '.join(bfo.search_pattern) - elif var == 'uid': - out = bfo.user_info['uid'] - elif var == 'referer': - out = bfo.user_info['referer'] - elif var == 'uri': - out = bfo.user_info['uri'] - elif var == 'nickname': - out = bfo.user_info['nickname'] - elif var == 'email': - out = bfo.user_info['email'] - else: - out = 'Unknown variable: %s' % (var) - - return out diff --git a/invenio/modules/formatter/format_elements/bfe_collection.py b/invenio/modules/formatter/format_elements/bfe_collection.py deleted file mode 100644 index bc7e62820..000000000 --- a/invenio/modules/formatter/format_elements/bfe_collection.py +++ /dev/null @@ -1,39 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints collection identifier -""" - -__revision__ = "$Id$" - -def format_element(bfo, kb): - """ - Prints the collection identifier. - Translate using given knowledge base. - - @param kb: a knowledge base use to translate the collection identifier - """ - - collection_identifiers = bfo.fields("980__a") - - for collection_identifier in collection_identifiers: - translated_collection_identifier = bfo.kb(kb, collection_identifier) - if translated_collection_identifier: - return translated_collection_identifier - - return '' diff --git a/invenio/modules/formatter/format_elements/bfe_comments.py b/invenio/modules/formatter/format_elements/bfe_comments.py deleted file mode 100644 index 6d4e94450..000000000 --- a/invenio/modules/formatter/format_elements/bfe_comments.py +++ /dev/null @@ -1,62 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints comments posted for the record -""" -__revision__ = "$Id$" - -from ...comments.api import get_first_comments_or_remarks - -def format_element(bfo, nbReviews='all', nbComments='all', show_reviews='true'): - """ - Prints comments posted for the record. - - @param nbReviews: The max number of reviews to print - @param nbComments: The max number of comments to print - @param show_reviews: Shows or hides the complete review block - """ - - nb_reviews = nbReviews - if nb_reviews.isdigit(): - nb_reviews = int(nb_reviews) - nb_comments = nbComments - if nb_comments.isdigit(): - nb_comments = int(nb_comments) - if show_reviews in ('true', 'True'): - show_reviews = True - else: - show_reviews = False - - (comments, reviews) = get_first_comments_or_remarks(recID=bfo.recID, - ln=bfo.lang, - nb_comments=nb_comments, - nb_reviews=nb_reviews, - voted=-1, - reported=-1, - user_info=bfo.user_info, - show_reviews=show_reviews) - - - return comments + reviews - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_contact.py b/invenio/modules/formatter/format_elements/bfe_contact.py deleted file mode 100644 index 1bf9c64e4..000000000 --- a/invenio/modules/formatter/format_elements/bfe_contact.py +++ /dev/null @@ -1,44 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints contact information -""" -__revision__ = "$Id$" - -def format_element(bfo, separator='; ', link="yes"): - """ - Prints contact information for the record - - @param separator: the separator between addresses. - @param link: Link the addresses to search engine (HTML links) if 'yes' - """ - from urllib import quote - from invenio.config import CFG_BASE_URL - - addresses = bfo.fields('270__p') - - if link == "yes": - addresses = ['' + \ - address +'' for address in addresses] - - return separator.join(addresses) - - - - diff --git a/invenio/modules/formatter/format_elements/bfe_copyright.py b/invenio/modules/formatter/format_elements/bfe_copyright.py deleted file mode 100644 index 1b14ab052..000000000 --- a/invenio/modules/formatter/format_elements/bfe_copyright.py +++ /dev/null @@ -1,794 +0,0 @@ -# -*- coding: utf-8 -*- -# -# $Id$ -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2012, 2013, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -from __future__ import print_function - -"""BibFormat element - Prints copyright information""" - -__revision__ = "$Id$" - -import sys - -if sys.hexversion < 0x2040000: - # pylint: disable=W0622 - from sets import Set as set - # pylint: enable=W0622 - -from six import iteritems - -from invenio.utils.url import create_html_link - -CFG_CERN_LICENSE_URL = 'http://copyright.cern.ch/' - -def format_element(bfo, copyrights_separator=", ", licenses_separator=", ", instances_separator=", ", - link_to_licenses='yes', auto_link_to_CERN_license='no', remove_link_to_CERN_license='yes', - show_licenses='yes', show_material="yes", - show_sponsor="yes", license_to_url_kb='LICENSE2URL'): - """ - Print copyright information - - Run this element to start the unit tests embedded within this element - - @param copyrights_separator: a separator between the copyrights of a bibdoc - @param licenses_separator: a separator between the licenses of a bibdoc - @param instances_separator: a separator between the licenses/copyrights of each BibDoc - @param link_to_licenses: if 'yes', print the links to the licenses (otherwise just the name) - @param auto_link_to_CERN_license: if 'yes', automatically add a link to CERN license when applicable (even if not in metadata) - @param remove_link_to_CERN_license: if 'yes', remove link to CERN license when existing in the metadata. This option is ignored when auto_link_to_CERN_license is set to 'yes' - @param show_licenses: if 'no', completely ignore license information - @param show_material: if 'yes', material to which license/copyright applies ($3 subfield) is displayed - @param license_to_url_kb: knowledge base used to map a license (as found in 540__a) to a URL - """ - if auto_link_to_CERN_license.lower() == 'yes': - # These option are mutually exclusive - remove_link_to_CERN_license = 'no' - add_CERN_license_link_to_bibdocs = [] - - copyrights_info = bfo.fields('542__', escape=1) - licenses_info = bfo.fields('540__', escape=1) - - copyrights_and_licenses_list = {} - if copyrights_info or licenses_info: - out = set() - - for copyright_info in copyrights_info: - - # Check to which bibdoc ID this copyright applies - bibdoc_id = 0 - material = None - if '8' in copyright_info: - try: - bibdoc_id = int(copyright_info['8']) - except: - pass - elif '3' in copyright_info: - # in that case, map using subfield $3 - material = copyright_info['3'] - - # Retrieve what to display to user - label = '' - if 'f' in copyright_info: - # Copyright message. Use this as label - label = copyright_info['f'] - elif 'd' in copyright_info: - # Copyright holder - year = '' - if 'g' in copyright_info: - # Year was given. Use it too - year = "%s " % copyright_info['g'] - label = "© " + year + copyright_info['d'] - if copyright_info['d'] == 'CERN' and \ - len(licenses_info) == 0 and \ - auto_link_to_CERN_license.lower() == 'yes': - # There is not license, it is a CERN copyright and - # we would like to add a link to the "license - # page" - add_CERN_license_link_to_bibdocs.append(bibdoc_id) - - elif 'e' in copyright_info: - # There is no copyright information available here? - # Display contact person - label = "Copyright info: " + copyright_info['e'] - else: - continue - - # Append our copyright to the list, for given BibDoc - if bibdoc_id not in copyrights_and_licenses_list and not (material and bibdoc_id == 0): - copyrights_and_licenses_list[bibdoc_id] = {'copyright':[], 'license': []} - elif material and material not in copyrights_and_licenses_list: - copyrights_and_licenses_list[material] = {'copyright':[], 'license': []} - if not (material and bibdoc_id == 0): - copyrights_and_licenses_list[bibdoc_id]['copyright'].append((label, copyright_info.get('d', ''))) - elif material in copyrights_and_licenses_list: - copyrights_and_licenses_list[material]['copyright'].append((label, copyright_info.get('d', ''))) - - # Now get the licenses. Try to map to a copyright - for license_info in licenses_info: - - # Check to which bibdoc ID this license applies - bibdoc_id = 0 - material = None - sponsor_info = license_info.get('f', '') - if license_info.has_key('8'): - try: - bibdoc_id = int(license_info['8']) - except: - pass - elif '3' in license_info: - # in that case, map using subfield $3 - material = license_info['3'] - label = '' - url = '' - license_body = '' - if 'a' in license_info: - # Terms governing use - label = license_info['a'] - if 'b' in license_info: - # Body imposing the license - license_body = license_info['b'] - if not url and \ - ((label in ('© CERN', 'CERN')) or ('CERN' in license_body)) and \ - auto_link_to_CERN_license.lower() == 'yes': - url = CFG_CERN_LICENSE_URL - if not url and license_to_url_kb: - # Look for URL in knowledge base - url = bfo.kb(license_to_url_kb, label) - if 'u' in license_info: - # License URL - url = license_info['u'] - - # Append our license to the list, for given BibDoc - if bibdoc_id not in copyrights_and_licenses_list and not (material and bibdoc_id == 0): - copyrights_and_licenses_list[bibdoc_id] = {'copyright':[], 'license': []} - elif material and material not in copyrights_and_licenses_list: - copyrights_and_licenses_list[material] = {'copyright':[], 'license': []} - if not (material and bibdoc_id == 0): - copyrights_and_licenses_list[bibdoc_id]['license'].append([label, license_body, url, sponsor_info]) - elif copyrights_and_licenses_list.has_key(material): - copyrights_and_licenses_list[material]['license'].append([label, license_body, url, sponsor_info]) - - # We also need to add the auto CERN licenses to specified BibDocs - for bibdoc_id in add_CERN_license_link_to_bibdocs: - copyrights_and_licenses_list[bibdoc_id]['license'].append(['', '', CFG_CERN_LICENSE_URL, '']) - - for linkage, copyright_and_license in iteritems(copyrights_and_licenses_list): - copyrights = copyright_and_license['copyright'] - licenses = copyright_and_license['license'] - if len(copyrights) == 1 and len(licenses) == 1: - # Great that is one particular case we can maybe handle - copyright_label, copyright_holder = copyrights[0] - license_label, license_body, license_url, dummy = licenses[0] - if not license_label or copyright_holder in ("© " + license_label, license_label): - # Cool, we can squash things - if remove_link_to_CERN_license.lower() == 'yes' and \ - license_url == CFG_CERN_LICENSE_URL: - # Thou must not display the license - license_url = '' - if show_material == 'yes' and not isinstance(linkage, int): - linkage_prefix = linkage + ': ' - else: - linkage_prefix = '' - if license_url and link_to_licenses.lower() == 'yes': - out.add(linkage_prefix + create_html_link(license_url, {}, copyright_label)) - else: - out.add(linkage_prefix + copyright_label) - continue - - # that is an 'else' for all other cases... - # First print simply the copyrights - copyright_tmp = copyrights_separator.join([copyright_label for (copyright_label, copyright_holder) in copyrights]) - if show_material == 'yes' and not isinstance(linkage, int): - copyright_tmp = linkage + ': ' + copyright_tmp - - license_tmp = [] - # Then do the licenses. - - # Check if we should prefix with "License(s):", i.e. when - # keyword license is missing from license labels - - prefix_license = '' - prefix_license_p = len([license_info for license_info in licenses if not 'license' in license_info[0].lower()]) - if prefix_license_p: - if len(licenses) == 1: - prefix_license = 'License: ' - else: - prefix_license = 'Licenses: ' - - sponsor_tmp = [] - for license_label, license_body, license_url, sponsor_info in licenses: - if sponsor_info: - sponsor_tmp.append(sponsor_info) - if not license_label and license_url and link_to_licenses.lower() == 'yes': - license_tmp.append(create_html_link(license_url, {}, 'License')) - prefix_license = '' # no longer needed - elif license_label and license_url and link_to_licenses.lower() == 'yes': - license_tmp.append(create_html_link(license_url, {}, license_label)) - elif license_label: - license_tmp.append(license_label) - - if sponsor_tmp and show_sponsor == "yes" and 'SCOAP3' in sponsor_tmp: - sponsored_by = 'sponsored by SCOAP³' - else: - sponsored_by = '' - - if show_licenses.lower() != 'yes': - # All that work for nothing! - license_tmp = [] - if copyright_tmp and license_tmp: - out.add('%s (%s%s)%s' % \ - (copyright_tmp, \ - prefix_license, \ - licenses_separator.join(license_tmp), \ - sponsored_by and ', %s' % sponsored_by or '' \ - )\ - ) - else: - out.add(copyright_tmp+licenses_separator.join(license_tmp)) - - return instances_separator.join(out).replace('©', '©') - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - -def test(): - """ - Test the function - """ - from invenio.modules.formatter.engine import BibFormatObject - - xml1 = ''' - - - CERN - -''' - - bfo1 = BibFormatObject(0, xml_record=xml1) - assert(format_element(bfo1) == '© CERN') - - - xml2 = ''' - - - CERN - ATLAS Experiment © CERN - -''' - - bfo2 = BibFormatObject(0, xml_record=xml2) - assert(format_element(bfo2) == 'ATLAS Experiment © CERN') - - - xml3 = ''' - - - BBC - -''' - - bfo3 = BibFormatObject(0, xml_record=xml3) - assert(format_element(bfo3) == '© BBC') - - - xml4 = ''' - - - CERN - http://cern.ch - - - CERN - -''' - - bfo4 = BibFormatObject(0, xml_record=xml4) - assert(format_element(bfo4) == '© CERN') - - - xml5 = ''' - - - BBC - http://bbc.co.uk - - - BBC - -''' - - bfo5 = BibFormatObject(0, xml_record=xml5) - assert(format_element(bfo5) == '© BBC') - - - xml6 = ''' - - - BBC - http://bbc.co.uk - - - CERN - -''' - - bfo6 = BibFormatObject(0, xml_record=xml6) - assert(format_element(bfo6) == '© CERN (License: BBC)') - - - xml7 = ''' - - - BBC - http://bbc.co.uk - - - 1 - CERN - -''' - - bfo7 = BibFormatObject(0, xml_record=xml7) - assert(format_element(bfo7) == '© CERN, BBC') - - - xml8 = ''' - - - BBC - http://bbc.co.uk - 1 - - - 1 - CERN - -''' - - bfo8 = BibFormatObject(0, xml_record=xml8) - assert(format_element(bfo8) == '© CERN (License: BBC)') - - - xml9 = ''' - - - BBC - http://bbc.co.uk - 1 - - - CERN - 2 - - - 2 - CERN - -''' - - bfo9 = BibFormatObject(0, xml_record=xml9) - assert(format_element(bfo9) == '© CERN, BBC') - - - xml10 = ''' - - - BBC - http://bbc.co.uk - 1 - - - CERN - 2 - - - 1 - BBC - - - 2 - CERN - -''' - - bfo10 = BibFormatObject(0, xml_record=xml10) - assert(format_element(bfo10) == '© BBC, © CERN') - - - xml11 = ''' - - - BBC License 1 - http://bbc.co.uk - 1 - - - CERN - 2 - - - 1 - BBC - - - 2 - CERN - -''' - - bfo11 = BibFormatObject(0, xml_record=xml11) - assert(format_element(bfo11) == '© CERN, © BBC (BBC License 1)') - - - xml12 = ''' - - - CERN - -''' - - bfo12 = BibFormatObject(0, xml_record=xml12) - assert(format_element(bfo12, auto_link_to_CERN_license='yes') == '© CERN' % CFG_CERN_LICENSE_URL) - - - xml13 = ''' - - - CERN - - - CERN - - - CERN - 2010 - -''' - - bfo13 = BibFormatObject(0, xml_record=xml13) - assert(format_element(bfo13, auto_link_to_CERN_license='yes') == '© CERN' % CFG_CERN_LICENSE_URL) - - -# xml14 = ''' -# -# -# CERN -# 2010 -# -# ''' - -# bfo14 = BibFormatObject(0, xml_record=xml14) -# assert(format_element(bfo14, auto_link_to_CERN_license='yes') == 'CERN') - - - xml15 = ''' - - - BBC License 1 - http://bbc.co.uk - 1 - - - CERN - 2 - - - 1 - BBC - - - 2 - CERN - -''' - - bfo15 = BibFormatObject(0, xml_record=xml15) - assert(format_element(bfo15, show_licenses='no', instances_separator=" & ") == '© BBC & © CERN') - - xml16 = ''' - - - BBC License 1 - http://bbc.co.uk - 1 - - - CERN - 2 - - - 1 - BBC - - - 2 - CERN - -''' - - bfo16 = BibFormatObject(0, xml_record=xml16) - assert(format_element(bfo16, link_to_licenses='no') == '© BBC (BBC License 1), © CERN') - - xml17 = ''' - - - BBC License 1 - http://bbc.co.uk/license1 - 1 - - - BBC License 2 - http://bbc.co.uk/license2 - 1 - - - CERN - 2 - - - 1 - BBC - - - 2 - CERN - -''' - - bfo17 = BibFormatObject(0, xml_record=xml17) - assert(format_element(bfo17) == '© CERN, © BBC (BBC License 1, BBC License 2)') - - xml18 = ''' - - - CERN - 1984 - -''' - - bfo18 = BibFormatObject(0, xml_record=xml18) - assert(format_element(bfo18) == '© 1984 CERN') - - xml19 = ''' - - - CERN - 1984 - ATLAS Experiment © CERN - -''' - - bfo19 = BibFormatObject(0, xml_record=xml19) - assert(format_element(bfo19) == 'ATLAS Experiment © CERN') - - - xml20 = ''' - - - BBC - 1984 - -''' - - bfo20 = BibFormatObject(0, xml_record=xml20) - assert(format_element(bfo20) == '© 1984 BBC') - - - xml21 = ''' - - - CERN - http://cern.ch - - - CERN - 1984 - -''' - - bfo21 = BibFormatObject(0, xml_record=xml21) - assert(format_element(bfo21) == '© 1984 CERN') - - - xml22 = ''' - - - BBC - http://bbc.co.uk - - - BBC - 1984 - -''' - - bfo22 = BibFormatObject(0, xml_record=xml22) - assert(format_element(bfo22) == '© 1984 BBC') - - - xml23 = ''' - - - CC-BY-3.0 - http://creativecommons.org/licenses/by/3.0/ - Preprint - - - CC-BY-3.0 - http://creativecommons.org/licenses/by/3.0/ - Publication - - - CERN - 2011 - Preprint - - - CERN - 2012 - Publication - -''' - - bfo23 = BibFormatObject(0, xml_record=xml23) - assert(format_element(bfo23) == 'Publication: © 2012 CERN (License: CC-BY-3.0), Preprint: © 2011 CERN (License: CC-BY-3.0)') - - - xml24 = ''' - - - CC-BY-3.0 - http://creativecommons.org/licenses/by/3.0/ - 1 - - - CC-BY-3.0 - http://creativecommons.org/licenses/by/3.0/ - Publication - - - CERN - 2011 - 1 - - - CERN - 2012 - Publication - -''' - - bfo24 = BibFormatObject(0, xml_record=xml24) - assert(format_element(bfo24) == 'Publication: © 2012 CERN (License: CC-BY-3.0), © 2011 CERN (License: CC-BY-3.0)') - - - xml25 = ''' - - - CC-BY-3.0 - http://creativecommons.org/licenses/by/3.0/ - Publication - - - CERN - Preprint - - - CERN - 2011 - Preprint - - - FOO - 2012 - Publication - -''' - - bfo25 = BibFormatObject(0, xml_record=xml25) - assert(format_element(bfo25, auto_link_to_CERN_license='yes') == 'Publication: © 2012 FOO (License: CC-BY-3.0), Preprint: © 2011 CERN') - - - xml26 = ''' - - - CC-BY-3.0 - Publication - - - CERN - Preprint - - - CERN - 2011 - Preprint - - - FOO - 2012 - Publication - -''' - - bfo26 = BibFormatObject(0, xml_record=xml26) - assert(format_element(bfo26, remove_link_to_CERN_license="no") == 'Publication: © 2012 FOO (License: CC-BY-3.0), Preprint: © 2011 CERN') - - xml27 = ''' - - - SCOAP3 - CC-BY-3.0 - Publication - - - CERN - Preprint - - - CERN - 2011 - Preprint - - - ESA - 2014 - Publication - -''' - - bfo27 = BibFormatObject(0, xml_record=xml27) - assert(format_element(bfo27, remove_link_to_CERN_license="yes") == 'Preprint: © 2011 CERN, Publication: © 2014 ESA (License: CC-BY-3.0), sponsored by SCOAP³') - - xml28 = ''' - - - SCOAP3 - CC-BY-3.0 - Publication - - - CERN - Preprint - - - CERN - 2011 - Preprint - - - ESA - 2014 - Publication - -''' - - bfo28 = BibFormatObject(0, xml_record=xml28) - assert(format_element(bfo28, remove_link_to_CERN_license="yes", show_sponsor="no") == 'Preprint: © 2011 CERN, Publication: © 2014 ESA (License: CC-BY-3.0)') - - print("All tests run ok") - -if __name__ == "__main__": - test() diff --git a/invenio/modules/formatter/format_elements/bfe_creation_date.py b/invenio/modules/formatter/format_elements/bfe_creation_date.py deleted file mode 100644 index 60ebd3ff9..000000000 --- a/invenio/modules/formatter/format_elements/bfe_creation_date.py +++ /dev/null @@ -1,40 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints creation date -""" -__revision__ = "$Id$" - -import datetime - -from invenio.modules.records.recordext.functions.get_creation_date import \ - get_creation_date - - -def format_element(bfo, format='%Y-%m-%d', date_format='%Y-%m-%d'): - ''' - Get the record creation date. - Note: parameter format is deprecated - - @param date_format: The date format in MySQL syntax - ''' - recID = bfo.recID - creation_date = get_creation_date(recID) or datetime.now() - - # Let's be gentle and backward compatible while "format" is here: - if date_format == '%Y-%m-%d' and format != '%Y-%m-%d': - date_format = format - return datetime.strptime(creation_date, date_format) diff --git a/invenio/modules/formatter/format_elements/bfe_dataset_appears.py b/invenio/modules/formatter/format_elements/bfe_dataset_appears.py deleted file mode 100644 index 40c496205..000000000 --- a/invenio/modules/formatter/format_elements/bfe_dataset_appears.py +++ /dev/null @@ -1,48 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the list of papers containing the dataset -""" -__revision__ = "$Id$" - - -def format_element(bfo): - """ - Prints the list of papers containing the dataset by title. - """ - - from invenio.modules.formatter.engine import BibFormatObject - from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD - - parent_recid = bfo.field("786__w") - bfo_parent = BibFormatObject(parent_recid) - - title = bfo_parent.field("245__a") - url = CFG_BASE_URL + '/' + CFG_SITE_RECORD + '/' + str(bfo_parent.recID) - - out = "This dataset complements the following publication:
" - out += "" + title + "" - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_date.py b/invenio/modules/formatter/format_elements/bfe_date.py deleted file mode 100644 index ed6a99bf9..000000000 --- a/invenio/modules/formatter/format_elements/bfe_date.py +++ /dev/null @@ -1,104 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints imprint publication date -""" -__revision__ = "$Id$" - -from invenio.utils.date import strftime, strptime, guess_datetime - -def format_element(bfo, date_format='%d %B %Y', source_formats='%Y-%m-%d', source_fields="260__c", - guess_source_format="no", ignore_date_format_for_year_only="yes"): - """ - Prints the imprint publication date. - - Parameter date_format allows to specify the string - representation of the output. - - The format string has the same behaviour as the strftime() function: - 
Eg: 1982-09-24 07:32:00
-            "%d %B %Y"   -> 24 September 1982
-            "%I:%M"      -> 07:32
-
- - Note that if input date is simply a year (4 digits), it is - returned as such if ignore_date_format_for_year_only - is set to 'yes', regardless of date_format. - - Parameter source_formats allows to specify the - expected format of the date in the metadata. If the format does - not match, the date cannot be parsed, and cannot be formatted - according to date_format. Comma-separated values can - be provided in order to test several input formats. - - Parameter source_fields defined the list of MARC - fields where we would like to retrieve the date. First one - matching source_formats is used. if none, fall back to - first non-empty one. - - Parameter guess_source_formats when set to 'yes' - allows to guess the date source format. - - - @see: pagination.py, publisher.py, reprints.py, imprint.py, place.py - @param date_format: output date format. - @param source_formats: expected (comma-separated values) input date format. - @param source_fields: the MARC fields (comma-separated values) to look up - for the date. First non-empty one is used. - @param guess_source_format: if 'yes', ignore 'source_format' and - try to guess format using Python mxDateTime module. - #param ignore_date_format_for_year_only: if 'yes', ignore 'date_format' when the - metadata in the record contains a single - year (4 digits). - """ - guess_source_format_p = guess_source_format.lower() == 'yes' - source_marc_fields = [source_marc_field.strip() for source_marc_field in source_fields.split(',')] - source_formats = [source_format.strip() for source_format in source_formats.split(',')] - ignore_date_format_for_year_only_p = ignore_date_format_for_year_only.lower() == 'yes' - parsed_datetime_value = None - first_matched_raw_date = '' - for source_marc_field in source_marc_fields: - date_value = bfo.field(source_marc_field) - if date_value: - if not first_matched_raw_date: - first_matched_raw_date = date_value - if ignore_date_format_for_year_only_p and \ - date_value.isdigit() and len(date_value) == 4: - # Year. Return as such - return date_value - if guess_source_format_p: - try: - parsed_datetime_value = guess_datetime(date_value) - break - except: - pass - else: - for source_format in source_formats: - try: - parsed_datetime_value = strptime(date_value, source_format) - break - except: - pass - if parsed_datetime_value: - # We have correctly parsed one date! - break - - if parsed_datetime_value: - return strftime(date_format, parsed_datetime_value) - else: - return first_matched_raw_date diff --git a/invenio/modules/formatter/format_elements/bfe_date_rec.py b/invenio/modules/formatter/format_elements/bfe_date_rec.py deleted file mode 100644 index 5201b6dbc..000000000 --- a/invenio/modules/formatter/format_elements/bfe_date_rec.py +++ /dev/null @@ -1,29 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints date of the entry of the record in database -""" -__revision__ = "$Id$" - -def format_element(bfo): - """ - Date of the entry of the record in the database - - @see: date.py - """ - date = bfo.field('909C1c') - - return date diff --git a/invenio/modules/formatter/format_elements/bfe_doi.py b/invenio/modules/formatter/format_elements/bfe_doi.py deleted file mode 100644 index 4139f4430..000000000 --- a/invenio/modules/formatter/format_elements/bfe_doi.py +++ /dev/null @@ -1,69 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2011, 2012 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints DOIs -""" - -from invenio.legacy.search_engine import get_field_tags -from cgi import escape - -def format_element(bfo, tag="909C4", label="", separator="
", description_location=""): - """ - Return an HTML link to the DOI. - - @param tag: field (tag + indicators) where the DOI can be found, if not specified, we take the tags asociated to the 'doi' logical field - @param separator: the separator between multiple tags - @param description_location: where should the description be added: if empty, the description is not printed; possible values: 'front', 'label', 'end' - @param label: label to use for the DOI link. If not specified, use the DOI number as label for the link. - """ - fields = [] - doi_tags = get_field_tags('doi') #first check the tags table - for doi_tag in doi_tags: - fields = bfo.fields(doi_tag[:5]) #we want only the tag, without the subfields - if fields: - break - if not fields: - fields = bfo.fields(tag) - doi_list = [] - for field in fields: - if field.get('2', 'DOI') == 'DOI' and 'a' in field: - desc = field.get('y', '') - front = end = '' - if desc: - if description_location == 'front': - front = desc + ': ' - elif description_location == 'label': - label = desc - elif description_location == 'end': - end = ' (' + desc + ')' - else: - front = end = '' - doi_list.append((field['a'], front, end, label)) - - if doi_list: - doi_link = """%s%s%s""" - return separator.join([doi_link % (escape(front), escape(doi, True), label and escape(label) or escape(doi), end) for (doi, front, end, label) in doi_list]) - else: - return "" - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_duration.py b/invenio/modules/formatter/format_elements/bfe_duration.py deleted file mode 100644 index 1169c71b1..000000000 --- a/invenio/modules/formatter/format_elements/bfe_duration.py +++ /dev/null @@ -1,81 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Display the duration of a media file -""" - -__revision__ = "$Id$" - -import re - -re_duration = re.compile('(\s|\A)(((?P\d{1,2}):)?(?P\d{1,2}):(?P\d\d))(\s|\Z)') - -def format_element(bfo, input_units="m", output_format="%(H)02i:%(M)02i:%(S)02i"): - """ - Display the duration of a media file (300__a). - - @param input_units: if field 300__a is digit, expect it to be: - - s: seconds - - m: minutes - - h: hours - If field 300__a is not digit, expect it to contain some variant of HH:MM:SS - @param output_format: how to display the duration. A basic Pythonic string replacement is - done on the input string. You can use the following keys: - - H: The number of hours - - M: The number of minutes - - S: The number of seconds - - H: The total number of hours - - M: The total number of minutes - - S: The total number of seconds - """ - # Get the duration - duration_in_seconds = None - duration = bfo.field('300__a') - try: - int_duration = int(float(duration)) - if input_units == 's': - duration_in_seconds = int_duration - elif input_units == 'm': - duration_in_seconds = int_duration * 60 - elif input_units == 'h': - duration_in_seconds = int_duration * 3600 - except Exception as e: - pass - - if duration_in_seconds is None: - # Try to find something like HH:MM:SS - match_obj = re_duration.match(duration) - if match_obj: - seconds = int(match_obj.group('seconds')) - minutes = int(match_obj.group('minutes')) - hours = match_obj.group('hours') - if not hours: - hours = 0 - else: - hours = int(hours) - duration_in_seconds = seconds + (60 * minutes) + (360 * hours) - - # Output the duration - if duration_in_seconds: - return output_format % {'H': duration_in_seconds / 3600, - 'M': (duration_in_seconds / 60) % 60, - 'S': duration_in_seconds % 60, - 'h': duration_in_seconds / 3600.0, # Absolute nb of hours - 'm': duration_in_seconds / 60.0, # Absolute nb of minutes - 's': duration_in_seconds #Absolute nb of seconds - } diff --git a/invenio/modules/formatter/format_elements/bfe_edit_files.py b/invenio/modules/formatter/format_elements/bfe_edit_files.py deleted file mode 100644 index 60d1ea621..000000000 --- a/invenio/modules/formatter/format_elements/bfe_edit_files.py +++ /dev/null @@ -1,59 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints a link to BibDocFile -""" -__revision__ = "$Id$" - -from invenio.utils.url import create_html_link -from invenio.base.i18n import gettext_set_language -from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD -from invenio.modules.access.engine import acc_authorize_action - -def format_element(bfo, style): - """ - Prints a link to simple file management interface (BibDocFile), if - authorization is granted. - - @param style: the CSS style to be applied to the link. - """ - _ = gettext_set_language(bfo.lang) - - out = "" - - user_info = bfo.user_info - (auth_code, auth_message) = acc_authorize_action(user_info, - 'runbibdocfile') - if auth_code == 0: - linkattrd = {} - if style != '': - linkattrd['style'] = style - - out += create_html_link(CFG_BASE_URL + '/%s/managedocfiles' % CFG_SITE_RECORD, - urlargd={'ln': bfo.lang, - 'recid': str(bfo.recID)}, - link_label=_("Manage Files of This Record"), - linkattrd=linkattrd) - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_edit_record.py b/invenio/modules/formatter/format_elements/bfe_edit_record.py deleted file mode 100644 index ef0a29c0c..000000000 --- a/invenio/modules/formatter/format_elements/bfe_edit_record.py +++ /dev/null @@ -1,60 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element to print a link to BibEdit.""" - -from invenio.base.i18n import gettext_set_language -from invenio.config import CFG_BASE_URL, CFG_SITE_RECORD -from invenio.legacy.bibedit.utils import user_can_edit_record_collection -from invenio.utils.url import create_html_link - - -def format_element(bfo, style, html_class='', link_label=None): - """Print a link to BibEdit, if authorization is granted. - - :param style: the CSS style to be applied to the link. - :param html_class: the class attribute to be applied to the link. - :param link_label: Localized link label. Default: "Edit This Record" - (or its translated variant). - """ - _ = gettext_set_language(bfo.lang) - - out = "" - - user_info = bfo.user_info - if user_can_edit_record_collection(user_info, bfo.recID): - linkattrd = {} - if style != '': - linkattrd['style'] = style - if html_class != '': - linkattrd['class'] = html_class - out += create_html_link( - CFG_BASE_URL + - '/%s/edit/?ln=%s#state=edit&recid=%s' % (CFG_SITE_RECORD, bfo.lang, - str(bfo.recID)), - {}, - link_label=link_label or _('Edit This Record'), - linkattrd=linkattrd) - - return out - - -def escape_values(bfo): - """Check if output of this element should be escaped.""" - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_editors.py b/invenio/modules/formatter/format_elements/bfe_editors.py deleted file mode 100644 index 8a52f3493..000000000 --- a/invenio/modules/formatter/format_elements/bfe_editors.py +++ /dev/null @@ -1,52 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints editors -""" -__revision__ = "$Id$" - -def format_element(bfo, limit, separator=' ; ', extension='[...]', print_links="yes"): - """ - Prints the list of editors of a record. - - @param limit: the maximum number of editors to display - @param separator: the separator between editors. - @param extension: a text printed if more editors than 'limit' exist - @param print_links: if yes, print the editors as HTML link to their publications - """ - from urllib import quote - from invenio.config import CFG_BASE_URL - from invenio.legacy import bibrecord - - authors = bibrecord.record_get_field_instances(bfo.get_record(), '100') - - editors = [bibrecord.field_get_subfield_values(author, 'a')[0] - for author in authors if len(bibrecord.field_get_subfield_values(author, "e")) > 0 and bibrecord.field_get_subfield_values(author, "e")[0]=="ed." ] - - if print_links.lower() == "yes": - editors = ['' + editor + '' - for editor in editors] - - if limit.isdigit() and len(editors) > int(limit): - return separator.join(editors[:int(limit)]) + extension - - elif len(editors) > 0: - return separator.join(editors) diff --git a/invenio/modules/formatter/format_elements/bfe_external_publications.py b/invenio/modules/formatter/format_elements/bfe_external_publications.py deleted file mode 100644 index ea5fcaffa..000000000 --- a/invenio/modules/formatter/format_elements/bfe_external_publications.py +++ /dev/null @@ -1,37 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints links to external publications -""" -__revision__ = "$Id$" - -def format_element(bfo, separator='
'): - """ - Prints list of links to external publications. - """ - publications = bfo.fields('909C4') - - out = [''+pub['p']+'' - for pub in publications - if 'd' in pub and 'p' in pub] - return separator.join(out) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_field.py b/invenio/modules/formatter/format_elements/bfe_field.py deleted file mode 100644 index 6babd9f14..000000000 --- a/invenio/modules/formatter/format_elements/bfe_field.py +++ /dev/null @@ -1,156 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints a custom field -""" -__revision__ = "$Id$" - -from six import iteritems -from invenio.modules.formatter.utils import parse_tag - -def format_element(bfo, tag, limit, instances_separator=" ", - subfields_separator=" ", extension="", output_pattern=""): - """ - Prints the given field of a record. - If tag is in range [001, 010], this element assumes - that it accesses a control field. Else it considers it - accesses a data field. - -

For eg. consider the following metdata: - 

- 100__ $$aCalatroni, S$$uCERN
- 245__ $$aStatus of the EP Simulations and Facilities for the SPL
- 700__ $$aFerreira, L$$uCERN
- 700__ $$aMacatrao, M$$uCERN
- 700__ $$aSkala, A$$uCERN
- 700__ $$aSosin, M$$uCERN
- 700__ $$ade Waele, R$$uCERN
- 700__ $$aWithofs, Y$$uKHLim, Diepenbeek
-
- The following calls to bfe_field would print: - 
-    <BFE_FIELD tag="700" instances_separator="<br/>" subfields_separator=" - ">
-
-    Ferreira, L - CERN
-    Macatrao, M - CERN
-    Skala, A - CERN
-    Sosin, M - CERN
-    de Waele, R - CERN
-    Withofs, Y - KHLim, Diepenbeek
-
-

- -

For more advanced formatting, the output_pattern - parameter can be used to output the subfields of each instance in - the specified way. For eg. consider the following metadata: - 

- 775__ $$b15. Aufl.$$c1995-1996$$nv.1$$pGrundlagen und Werkstoffe$$w317999
- 775__ $$b12. Aufl.$$c1963$$w278898
- 775__ $$b14. Aufl.$$c1983$$w107899
- 775__ $$b13. Aufl.$$c1974$$w99635
-
- with the following output_pattern: - - 
-    <a href="/record/%(w)s">%(b)s (%(c)s) %(n)s %(p)s</a>
-
- would print:
- - 15. Aufl. (1995-1996) v.1 Grundlagen und Werkstoffe
- 12. Aufl. (1963)
- 14. Aufl. (1983)
- 13. Aufl. (1974) - -
(instances_separator="<br/>" set for - readability)
The output pattern must follow Python - string formatting syntax. The format must use parenthesized - notation to map to the subfield code. This currently restricts the - support of output_pattern to non-repeatable - subfields

- - @param tag: the tag code of the field that is to be printed - @param instances_separator: a separator between instances of field - @param subfields_separator: a separator between subfields of an instance - @param limit: the maximum number of values to display. - @param extension: a text printed at the end if 'limit' has been exceeded - @param output_pattern: when specified, prints the subfields of each instance according to pattern specified as parameter (following Python string formatting convention) - """ - # Check if data or control field - p_tag = parse_tag(tag) - if p_tag[0].isdigit() and int(p_tag[0]) in range(0, 11): - return bfo.control_field(tag) - elif p_tag[0].isdigit(): - # Get values without subcode. - # We will filter unneeded subcode later - if p_tag[1] == '': - p_tag[1] = '_' - if p_tag[2] == '': - p_tag[2] = '_' - values = bfo.fields(p_tag[0]+p_tag[1]+p_tag[2]) # Values will - # always be a - # list of - # dicts - else: - return '' - - x = 0 - instances_out = [] # Retain each instance output - for instance in values: - filtered_values = [value for (subcode, value) in iteritems(instance) - if p_tag[3] == '' or p_tag[3] == '%' \ - or p_tag[3] == subcode] - if len(filtered_values) > 0: - # We have found some corresponding subcode(s) - if limit.isdigit() and x + len(filtered_values) >= int(limit): - # We are going to exceed the limit - filtered_values = filtered_values[:int(limit)-x] # Takes only needed one - if len(filtered_values) > 0: # do not append empty list! - if output_pattern: - try: - instances_out.append(output_pattern % DictNoKeyError(instance)) - except: - pass - else: - instances_out.append(subfields_separator.join(filtered_values)) - x += len(filtered_values) # record that so we know limit has been exceeded - break # No need to go further - else: - if output_pattern: - try: - instances_out.append(output_pattern % DictNoKeyError(instance)) - except: - pass - else: - instances_out.append(subfields_separator.join(filtered_values)) - x += len(filtered_values) - - ext_out = '' - if limit.isdigit() and x > int(limit): - ext_out = extension - - return instances_separator.join(instances_out) + ext_out - - -class DictNoKeyError(dict): - def __getitem__(self, key): - if dict.__contains__(self, key): - val = dict.__getitem__(self, key) - else: - val = '' - return val diff --git a/invenio/modules/formatter/format_elements/bfe_fulltext.py b/invenio/modules/formatter/format_elements/bfe_fulltext.py deleted file mode 100644 index 291239d48..000000000 --- a/invenio/modules/formatter/format_elements/bfe_fulltext.py +++ /dev/null @@ -1,367 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints a links to fulltext -""" -__revision__ = "$Id$" - -import re -from invenio.legacy.bibdocfile.api import BibRecDocs, file_strip_ext, normalize_format, compose_format -from invenio.base.i18n import gettext_set_language -from invenio.config import CFG_SITE_URL, CFG_BASE_URL, CFG_CERN_SITE, CFG_SITE_RECORD, \ - CFG_BIBFORMAT_HIDDEN_FILE_FORMATS -from invenio.legacy.bibdocfile.config import CFG_BIBDOCFILE_ICON_SUBFORMAT_RE -from invenio.utils.url import get_relative_url - -from cgi import escape, parse_qs -from six.moves.urllib.parse import urlparse -from os.path import basename -import urllib - -_CFG_NORMALIZED_BIBFORMAT_HIDDEN_FILE_FORMATS = set(normalize_format(fmt) for fmt in CFG_BIBFORMAT_HIDDEN_FILE_FORMATS) - -_CFG_BIBFORMAT_HIDDEN_DOCTYPES = ['Plot'] -if CFG_CERN_SITE: - _CFG_BIBFORMAT_HIDDEN_DOCTYPES.append('arXiv') - -cern_arxiv_categories = ["astro-ph", "chao-dyn", "cond-mat", "gr-qc", - "hep-ex", "hep-lat", "hep-ph", "hep-th", "math-ph", - "math", "nucl-ex", "nucl-th", "out", "physics", - "quant-ph", "q-alg", "cs", "adap-org", "comp-gas", - "chem-ph", "cs", "math", "neuro-sys", "patt-sol", - "solv-int", "acc-phys", "alg-geom", "ao-sci", - "atom-ph", "cmp-lg", "dg-ga", "funct-an", "mtrl-th", - "plasm-ph", "q-alg", "supr-con"] - -def format_element(bfo, style, separator='; ', show_icons='no', focus_on_main_file='no', show_subformat_icons='no'): - """ - This is the default format for formatting fulltext links. - - When possible, it returns only the main file(s) (+ link to - additional files if needed). If no distinction is made at - submission time between main and additional files, returns - all the files - - @param separator: the separator between urls. - @param style: CSS class of the link - @param show_icons: if 'yes', print icons for fulltexts - @param focus_on_main_file: if 'yes' and a doctype 'Main' is found, - prominently display this doctype. In that case other doctypes are - summarized with a link to the Files tab, named "Additional files" - @param show_subformat_icons: shall we display subformats considered as icons? - """ - _ = gettext_set_language(bfo.lang) - - out = '' - - # Retrieve files - (parsed_urls, old_versions, additionals) = get_files(bfo, \ - distinguish_main_and_additional_files=focus_on_main_file.lower() == 'yes', - include_subformat_icons=show_subformat_icons == 'yes', - hide_doctypes=_CFG_BIBFORMAT_HIDDEN_DOCTYPES) - - main_urls = parsed_urls['main_urls'] - others_urls = parsed_urls['others_urls'] - if 'cern_urls' in parsed_urls: - cern_urls = parsed_urls['cern_urls'] - - # Prepare style and icon - if style != "": - style = 'class="'+style+'"' - - if show_icons.lower() == 'yes': - file_icon = '%s' % (CFG_BASE_URL, _("Download fulltext")) - else: - file_icon = '' - - # Build urls list. - # Escape special chars for tag value. - - additional_str = '' - if additionals: - additional_str = ' (%s)' % _("additional files") - - versions_str = '' - #if old_versions: - #versions_str = ' (%s)' % _("older versions") - - if main_urls: - out = [] - main_urls_keys = sort_alphanumerically(main_urls.keys()) - for descr in main_urls_keys: - urls = main_urls[descr] - if re.match(r'^\d+\s', descr) and urls[0][2] == 'png': - # FIXME: we have probably hit a Plot (as link - # description looks like '0001 This is Caption'), so - # do not take it. This test is not ideal, we should - # rather study doc type, and base ourselves on - # Main/Additional/Plot etc. - continue - out += ['
  • %s:
    • ' % descr] - urls_dict = {} - for url, name, url_format in urls: - if name not in urls_dict: - urls_dict[name] = [(url, url_format)] - else: - urls_dict[name].append((url, url_format)) - for name, urls_and_format in urls_dict.items(): - if len(urls_dict) > 1: - print_name = "%s" % name - url_list = ['
  • ' + print_name + "
    • "] - else: - url_list = [] - for url, url_format in urls_and_format: - if CFG_CERN_SITE and url_format == 'ps.gz' and len(urls_and_format) > 1: - ## We skip old PS.GZ files - continue - url_list.append('
  • %(file_icon)s %(url_format)s
    • ' % { - 'style': style, - 'url': escape(url, True), - 'file_icon': file_icon, - 'url_format': escape(url_format.upper()) - }) - out += url_list - return '
      ' + "\n".join(out) + '
    ' - - if main_urls: - main_urls_keys = sort_alphanumerically(main_urls.keys()) - for descr in main_urls_keys: - urls = main_urls[descr] - out += "%s: " % descr - urls_dict = {} - for url, name, url_format in urls: - if name not in urls_dict: - urls_dict[name] = [(get_relative_url(url), url_format)] - else: - urls_dict[name].append((get_relative_url(url), url_format)) - for name, urls_and_format in urls_dict.items(): - if len(urls_dict) > 1: - print_name = "%s - " % name - url_list = [print_name] - else: - url_list = [] - for url, url_format in urls_and_format: - if CFG_CERN_SITE and url_format == 'ps.gz' and len(urls_and_format) > 1: - ## We skip old PS.GZ files - continue - url_list.append('%(file_icon)s%(url_format)s' % { - 'style': style, - 'url': escape(url, True), - 'file_icon': file_icon, - 'url_format': escape(url_format.upper()) - }) - out += " ".join(url_list) + additional_str + versions_str + separator - - if CFG_CERN_SITE and cern_urls: - link_word = len(cern_urls) == 1 and _('%(x_sitename)s link') or _('%(x_sitename)s links') - out += '%s: ' % (link_word % {'x_sitename': 'CERN'}) - url_list = [] - for url, descr in cern_urls: - url_list.append(''+ \ - file_icon + escape(str(descr))+'') - out += separator.join(url_list) - - if others_urls: - external_link = len(others_urls) == 1 and _('external link') or _('external links') - out += '%s: ' % external_link.capitalize() - url_list = [] - for url, descr in others_urls: - url_list.append(''+ \ - file_icon + escape(str(descr))+'') - out += separator.join(url_list) + '
    ' - - if out.endswith('
    '): - out = out[:-len('
    ')] - - # When exported to text (eg. in WebAlert emails) we do not want to - # display the link to the fulltext: - if out: - out = '' + out + '' - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - -def get_files(bfo, distinguish_main_and_additional_files=True, include_subformat_icons=False, - hide_doctypes=None): - """ - Returns the files available for the given record. - Returned structure is a tuple (parsed_urls, old_versions, additionals): - - parsed_urls: contains categorized URLS (see details below) - - old_versions: set to True if we can have access to old versions - - additionals: set to True if we have other documents than the 'main' document - - Parameter 'include_subformat_icons' decides if subformat - considered as icons should be returned - - Parameter hide_doctypes (list) decides which doctypes should not - be included in the returned structure - - 'parsed_urls' is a dictionary in the form:: - {'main_urls' : {'Main' : [('http://CFG_SITE_URL/CFG_SITE_RECORD/1/files/aFile.pdf', 'aFile', 'PDF'), - ('http://CFG_SITE_URL/CFG_SITE_RECORD/1/files/aFile.gif', 'aFile', 'GIF')], - 'Additional': [('http://CFG_SITE_URL/CFG_SITE_RECORD/1/files/bFile.pdf', 'bFile', 'PDF')]}, - - 'other_urls': [('http://externalurl.com/aFile.pdf', 'Fulltext'), # url(8564_u), description(8564_z/y) - ('http://externalurl.com/bFile.pdf', 'Fulltext')], - - 'cern_urls' : [('http://cern.ch/aFile.pdf', 'Fulltext'), # url(8564_u), description(8564_z/y) - ('http://cern.ch/bFile.pdf', 'Fulltext')], - } - - Some notes about returned structure: - - key 'cern_urls' is only available on CERN site - - keys in main_url dictionaries are defined by the BibDoc. - - older versions are not part of the parsed urls - - returns only main files when possible, that is when doctypes - make a distinction between 'Main' files and other - files. Otherwise returns all the files as main. This is only - enabled if distinguish_main_and_additional_files is set to True - """ - - _ = gettext_set_language(bfo.lang) - - if hide_doctypes is None: - hide_doctypes = [] - - urls = bfo.fields("8564_") - bibarchive = BibRecDocs(bfo.recID) - - old_versions = False # We can provide link to older files. Will be - # set to True if older files are found. - additionals = False # We have additional files. Will be set to - # True if additional files are found. - - # Prepare object to return - parsed_urls = {'main_urls':{}, # Urls hosted by Invenio (bibdocs) - 'others_urls':[] # External urls - } - if CFG_CERN_SITE: - parsed_urls['cern_urls'] = [] # cern.ch urls - - if [url for url in urls if url.get('u', '').startswith('http://arxiv.org/pdf/')]: - # We have a link to arXiv PDF. We can hide the files on - # CDS in some cases: - hide_doctypes.append('CMSPUB_SOURCEF') - hide_doctypes.append('ATLPUB_SOURCEF') - hide_doctypes.append('LHCBPB_SOURCEF') - - # Doctypes can of any type, but when there is one file marked as - # 'Main', we consider that there is a distinction between "main" - # and "additional" files. Otherwise they will all be considered - # equally as main files - distinct_main_and_additional_files = False - if len(bibarchive.list_bibdocs(doctype='Main')) > 0 and \ - distinguish_main_and_additional_files: - distinct_main_and_additional_files = True - # Parse URLs - for complete_url in urls: - if 'u' in complete_url: - url = complete_url['u'] - (dummy, host, path, dummy, params, dummy) = urlparse(url) - subformat = complete_url.get('x', '') - filename = urllib.unquote(basename(path)) - name = file_strip_ext(filename) - url_format = filename[len(name):] - if url_format.startswith('.'): - url_format = url_format[1:] - if compose_format(url_format, subformat) in _CFG_NORMALIZED_BIBFORMAT_HIDDEN_FILE_FORMATS: - ## This format should be hidden. - continue - - descr = _("Fulltext") - if 'y' in complete_url: - descr = complete_url['y'] - if descr == 'Fulltext': - descr = _("Fulltext") - if not url.startswith(CFG_SITE_URL): # Not a bibdoc? - if not descr: # For not bibdoc let's have a description - # Display the URL in full: - descr = url - if CFG_CERN_SITE and 'cern.ch' in host and \ - ('/setlink?' in url or \ - 'cms' in host or \ - 'documents.cern.ch' in url or \ - 'doc.cern.ch' in url or \ - 'preprints.cern.ch' in url): - url_params_dict = dict([part.split('=') for part in params.split('&') if len(part.split('=')) == 2]) - if 'categ' in url_params_dict and \ - (url_params_dict['categ'].split('.', 1)[0] in cern_arxiv_categories) and \ - 'id' in url_params_dict: - # Old arXiv links, used to be handled by - # setlink. Provide direct links to arXiv - for file_format, label in [('pdf', "PDF")]:#, - #('ps', "PS"), - #('e-print', "Source (generally TeX or LaTeX)"), - #('abs', "Abstract")]: - url = "http://arxiv.org/%(format)s/%(category)s/%(id)s" % \ - {'format': file_format, - 'category': url_params_dict['categ'], - 'id': url_params_dict['id']} - parsed_urls['others_urls'].append((url, "%s/%s %s" % \ - (url_params_dict['categ'], - url_params_dict['id'], - label))) - else: - parsed_urls['others_urls'].append((url, descr)) # external url - else: # It's a bibdoc! - assigned = False - for doc in bibarchive.list_bibdocs(): - if int(doc.get_latest_version()) > 1: - old_versions = True - if True in [f.get_full_name().startswith(filename) \ - for f in doc.list_all_files()]: - assigned = True - if not include_subformat_icons and \ - CFG_BIBDOCFILE_ICON_SUBFORMAT_RE.match(subformat): - # This is an icon and we want to skip it - continue - doctype = doc.get_doctype(bfo.recID) - if doctype in hide_doctypes: - continue - if not doctype == 'Main' and \ - distinct_main_and_additional_files == True: - # In that case we record that there are - # additional files, but don't add them to - # returned structure. - additionals = True - else: - if not descr: - descr = _('Fulltext') - if descr not in parsed_urls['main_urls']: - parsed_urls['main_urls'][descr] = [] - params_dict = parse_qs(params) - if 'subformat' in params_dict: - url_format += ' (%s)' % params_dict['subformat'][0] - parsed_urls['main_urls'][descr].append((url, name, url_format)) - if not assigned: # Url is not a bibdoc :-S - if not descr: - descr = filename - parsed_urls['others_urls'].append((url, descr)) # Let's put it in a general other url - return (parsed_urls, old_versions, additionals) - -_RE_SPLIT = re.compile(r"\d+|\D+") -def sort_alphanumerically(elements): - elements = [([not token.isdigit() and token or int(token) for token in _RE_SPLIT.findall(element)], element) for element in elements] - elements.sort() - return [element[1] for element in elements] diff --git a/invenio/modules/formatter/format_elements/bfe_fulltext_mini.py b/invenio/modules/formatter/format_elements/bfe_fulltext_mini.py deleted file mode 100644 index d6f6a524c..000000000 --- a/invenio/modules/formatter/format_elements/bfe_fulltext_mini.py +++ /dev/null @@ -1,156 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints a links to fulltext -""" -__revision__ = "$Id$" - -from invenio.modules.formatter.format_elements.bfe_fulltext import get_files, sort_alphanumerically, _CFG_BIBFORMAT_HIDDEN_DOCTYPES -from invenio.base.i18n import gettext_set_language -from invenio.config import CFG_SITE_URL, CFG_BASE_URL, CFG_CERN_SITE, CFG_SITE_RECORD -from invenio.utils.url import get_relative_url -from cgi import escape - -def format_element(bfo, style, separator='; ', show_icons='no', focus_on_main_file='yes', show_subformat_icons='no'): - """ - This is the format for formatting fulltext links in the mini panel. - @param separator: the separator between urls. - @param style: CSS class of the link - @param show_icons: if 'yes', print icons for fulltexts - @param focus_on_main_file: if 'yes' and a doctype 'Main' is found, - prominently display this doctype. In that case other doctypes are - summarized with a link to the Files tab, named"Additional files". - @param show_subformat_icons: shall we display subformats considered as icons? - """ - _ = gettext_set_language(bfo.lang) - out = '' - - # Retrieve files - (parsed_urls, old_versions, additionals) = \ - get_files(bfo, distinguish_main_and_additional_files=focus_on_main_file.lower() == 'yes', - include_subformat_icons=show_subformat_icons == 'yes', - hide_doctypes=_CFG_BIBFORMAT_HIDDEN_DOCTYPES) - - main_urls = parsed_urls['main_urls'] - others_urls = parsed_urls['others_urls'] - if 'cern_urls' in parsed_urls: - cern_urls = parsed_urls['cern_urls'] - - # Prepare style - if style != "": - style = 'class="'+style+'"' - - # Build urls list. - # Escape special chars for tag value. - - additional_str = '' - if additionals: - additional_str = separator + '(%s)' % _("additional files") - - versions_str = '' - #if old_versions: - #versions_str = separator + '(%s)' % _("older versions") - - if main_urls: - # Put a big file icon if only one file - if len(main_urls.keys()) == 1 and len(main_urls.items()[0][1]) == 1 and \ - (not CFG_CERN_SITE or len(cern_urls) == 0) and len(others_urls) == 0 and \ - show_icons.lower() == 'yes': - file_icon = '%s
    ' % (CFG_BASE_URL, _("Download fulltext")) - - elif show_icons.lower() == 'yes': - file_icon = '%s' % (CFG_BASE_URL, _("Download fulltext")) - else: - file_icon = '' - - main_urls_keys = sort_alphanumerically(main_urls.keys()) - for descr in main_urls_keys: - urls = main_urls[descr] - out += '
    %s: ' % descr - urls_dict = {} - for url, name, url_format in urls: - if name not in urls_dict: - urls_dict[name] = [(url, url_format)] - else: - urls_dict[name].append((url, url_format)) - for name, urls_and_format in urls_dict.items(): - if len(urls_dict) > 1: - print_name = "%s - " % name - url_list = [print_name] - else: - url_list = [] - for url, url_format in urls_and_format: - if CFG_CERN_SITE and url_format == 'ps.gz' and len(urls_and_format) > 1: - ## We skip old PS.GZ files - continue - url_list.append('%(file_icon)s%(url_format)s' % { - 'style': style, - 'url': get_relative_url(escape(url, True)), - 'file_icon': file_icon, - 'url_format': escape(url_format.upper()) - }) - out += separator + " ".join(url_list) - out += additional_str + versions_str + separator + "
    " - - if CFG_CERN_SITE and cern_urls: - # Put a big file icon if only one file - if len(main_urls.keys()) == 0 and \ - len(cern_urls) == 1 and len(others_urls) == 0 and \ - show_icons.lower() == 'yes': - file_icon = '%s
    ' % (CFG_BASE_URL, _("Download fulltext")) - - elif show_icons.lower() == 'yes': - file_icon = '%s' % (CFG_BASE_URL, _("Download fulltext")) - else: - file_icon = '' - - link_word = len(cern_urls) == 1 and _('%(x_sitename)s link') or _('%(x_sitename)s links') - out += '%s:
    ' % (link_word % {'x_sitename': 'CERN'}) - url_list = [] - for url, descr in cern_urls: - url_list.append(''+file_icon+escape(str(descr))+'') - out += '' + separator.join(url_list) + '' - out += "
    " - - if others_urls: - # Put a big file icon if only one file - if len(main_urls.keys()) == 0 and \ - (not CFG_CERN_SITE or len(cern_urls) == 0) and len(others_urls) == 1 and \ - show_icons.lower() == 'yes': - file_icon = '%s
    ' % (CFG_BASE_URL, _("Download fulltext")) - elif show_icons.lower() == 'yes': - file_icon = '%s' % (CFG_BASE_URL, _("Download fulltext")) - else: - file_icon = '' - external_link = len(others_urls) == 1 and _('external link') or _('external links') - out += '%s:%s' % (external_link.capitalize(), separator) - url_list = [] - for url, descr in others_urls: - url_list.append(''+file_icon+escape(str(descr))+'') - out += '' + separator.join(url_list) + '' - if out.endswith('
    '): - out = out[:-len('
    ')] - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_hepdata_table.py b/invenio/modules/formatter/format_elements/bfe_hepdata_table.py deleted file mode 100644 index 0a5c826e4..000000000 --- a/invenio/modules/formatter/format_elements/bfe_hepdata_table.py +++ /dev/null @@ -1,48 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints HEP Data table encoded by a record -""" -__revision__ = "$Id$" - -from invenio.utils.hepdata import display as hepdatadisplayutils -from invenio.utils import hepdata as hepdatautils -def format_element(bfo): - """ - Prints HEPData table encoded in the record - """ - publisher = bfo.fields("520__9")[0] - if publisher == "HEPDATA": - recid = bfo.recID - parent_recid = int(bfo.fields("786__w")[0]) - seq = int(bfo.fields("786__q")[0]) - dataset = hepdatautils.get_hepdata_by_recid(parent_recid, recid) - return hepdatadisplayutils.render_hepdata_dataset_html(dataset, parent_recid, seq, display_link = False) - elif publisher == "Dataverse": - return hepdatadisplayutils.render_dataverse_dataset_html(bfo.recID, display_link = False) - elif publisher == "INSPIRE": - return hepdatadisplayutils.render_inspire_dataset_html(bfo.recID, display_link = False) - else: - return hepdatadisplayutils.render_other_dataset_html(bfo.recID, display_link = False) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_imprint.py b/invenio/modules/formatter/format_elements/bfe_imprint.py deleted file mode 100644 index 56294a4d2..000000000 --- a/invenio/modules/formatter/format_elements/bfe_imprint.py +++ /dev/null @@ -1,65 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints document imprint -""" -__revision__ = "$Id$" - -from invenio.utils.date import strftime, strptime - -def format_element(bfo, place_label, publisher_label, date_label, - separator=', ', date_format=""): - """ - Print imprint (Order: Name of publisher, place of publication and date of publication). - Parameter date_format allows to specify the string representation of the output. - The format string has the same behaviour as the strftime() function:: - 
    Eg: 1982-09-24 07:32:00
-             "%d %B %Y"   -> 24 September 1982
-             "%I:%M"      -> 07:32
-
    - @param separator: a separator between the elements of imprint - @param place_label: a label to print before the publication place value - @param publisher_label: a label to print before the publisher name - @param date_label: a a label to print before the publication date - @param date_format: date format - @see: place.py, publisher.py, date.py, reprints.py, pagination.py - """ - - place = bfo.field('260__a') - publisher = bfo.field('260__b') - date = bfo.field('260__c') - - out = "" - - if publisher != "sine nomine": - out += publisher_label + ' ' + publisher + separator - - if place != "sine loco": - out += place_label + ' ' + place + separator - - if len(date) > 0: - if date_format != '': - try: - date_time = strptime(date, "%Y-%m-%d") - out += date_label + " " + strftime(date_format, date_time) - except ValueError: - out += date_label + ' ' + date - else: - out += date_label + ' ' + date - - return out diff --git a/invenio/modules/formatter/format_elements/bfe_isbn.py b/invenio/modules/formatter/format_elements/bfe_isbn.py deleted file mode 100644 index ea0117ff2..000000000 --- a/invenio/modules/formatter/format_elements/bfe_isbn.py +++ /dev/null @@ -1,40 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Print ISBN -""" - -__revision__ = "$Id$" - -def format_element(bfo, separator=", "): - """ - Returns the ISBN of the record. - """ - isbn_numbers = bfo.fields("020__a") - - return separator.join(isbn_numbers) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - - diff --git a/invenio/modules/formatter/format_elements/bfe_issn.py b/invenio/modules/formatter/format_elements/bfe_issn.py deleted file mode 100644 index 1aabe6d53..000000000 --- a/invenio/modules/formatter/format_elements/bfe_issn.py +++ /dev/null @@ -1,1086 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Print ISSN corresponding to given journal name -""" - -from __future__ import print_function - -__revision__ = "$Id$" - -import pprint -import sys -import re -import getopt -from invenio.legacy.search_engine import perform_request_search -from invenio.legacy.bibrecord import get_fieldvalues -from invenio.config import CFG_CERN_SITE -from invenio.utils.url import make_invenio_opener - -BIBFORMAT_OPENER = make_invenio_opener('BibFormat') - -if CFG_CERN_SITE: - journal_name_tag = '773__p' -else: - journal_name_tag = '909C4p' - -issns = { 'aapps bull.': '0218-2203', - 'account. manag. inf. technol.': '0959-8022', - 'acm comput. surv.': '0360-0300', - 'acm sigplan not.': '0362-1340', - 'acm trans. comput. syst.': '0734-2071', - 'acm trans. comput.-hum. interact.': '1073-0516', - 'acm trans. database syst.': '0362-5915', - 'acm trans. graph.': '0730-0301', - 'acm trans. inf. syst. secur.': '1094-9224', - 'acm trans. internet technol.': '1533-5399', - 'acm trans. math. softw.': '0098-3500', - 'acm trans. program. lang. syst.': '0164-0925', - 'acm trans. storage': '1553-3077', - 'acta appl. math.': '1572-9036', - 'acta arith.': '1730-6264', - 'acta fac. rerum nat. univ. comen.: math.': '0373-8183', - 'acta math. appl. sin.': '1618-3932', - 'acta math. hung.': '1588-2632', - 'acta mech. sin.': '1614-3116', - 'acta mech. solida sin.': '1860-2134', - 'acta phys. pol. a': '0587-4246', - 'ad hoc netw.': '1570-8705', - 'adsorption': '1572-8757', - 'adv. cement based mater.': '1065-7355', - 'adv. colloid interface sci.': '0001-8686', - 'adv. compos. mater.': '1568-5519', - 'adv. eng. inform.': '1474-0346', - 'adv. eng. mater.': '1527-2648', - 'adv. geom.': '1615-715X', - 'adv. mater.': '1521-4095', - 'adv. math.': '0001-8708', - 'adv. perform. mater.': '1572-8765', - 'adv. powder technol.': '1568-5527', - 'adv. robot.': '1568-5535', - 'air space eur.': '1290-0958', - 'algebr. represent. theory': '1572-9079', - 'anal. math.': '0133-3852', - 'analog integr. circuits signal process.': '0925-1030', - 'angew. chem.': '1521-3757', - 'angew. chem. int. ed.': '1521-3773', - 'ann. glob. anal. geom.': '0232-704X', - 'ann. inst. stat. math.': '0020-3157', - 'ann. mat. pura appl.': '1618-1891', - 'ann. math. artif. intell.': '1012-2443', - 'ann. oper. res.': '1572-9338', - 'ann. phys. (san diego)': '0003-4916', - 'ann. phys. (weinheim)': '1521-3889', - 'ann. pol. math.': '1730-6272', - 'ann. sci. ec. norm. sup\xc3\xa9r.': '0012-9593', - 'ann. softw. eng.': '1573-7489', - 'annu. rev. nucl. part. sci.': '0163-8998', - 'appl. compos. mater.': '0929-189X', - 'appl. intell.': '0924-669X', - 'appl. math.': '0862-7940', - 'appl. math. electron. notes': '1607-2510', - 'appl. phys.': '0340-3793', - 'appl. soft comput.': '1568-4946', - 'appl. stoch. models bus. ind.': '1526-4025', - 'appl. supercond.': '0964-1807', - 'appl. surf. sci.': '0378-5963', - 'approx. theory. appl.': '1000-9221', - 'arch. mus. inform.': '1573-7500', - 'arch. sci.': '1573-7519', - 'ariadne': '1361-3200', - 'artif. intell. rev.': '0269-2821', - 'artif. life robot.': '1614-7456', - 'asimmetrie': '1827-1383', - 'astron. educ. rev.': '1539-1515', - 'astron. q.': '0364-9229', - 'astrophys. space sci.': '1572-946X', - 'astrophys. space sci. trans.': '1810-6536', - 'astrophysics': '1573-8191', - 'at. data nucl. data tables': '0092-640X', - 'at. energy': '1573-8205', - 'atom': '0004-7015', - 'autom. remote control.': '0005-1179', - 'autom. softw. eng.': '0928-8910', - 'auton. agents multi-agent syst.': '1387-2532', - 'auton. robots': '0929-5593', - 'banach cent. publ.': '1730-6299', - 'ber. bunsenges. phys. chem.': '0005-9021', - 'ber. wiss.gesch.': '1522-2365', - 'bioelectromagnetics': '1521-186X', - 'biom. j.': '1521-4036', - 'biomed. eng.': '0006-3398', - 'biophys. chem.': '0301-4622', - 'biosens. bioelecton.': '0956-5663', - 'bol. asoc. math. venez.': '1315-4125', - 'br. j. appl. phys.': '0508-3443', - 'bt technol. j.': '1358-3948', - 'bulg. j. phys.': '1310-0157', - 'bull. earthq. eng.': '1573-1456', - 'bull. soc. math. fr.': '0037-9484', - 'bull. union phys.': '0366-3878', - 'bus. strategy environ.': '1099-0836', - 'c. r. acad. sci., 2b': '1620-7742', - 'c. r. mech.': '1631-0721', - 'c. r. phys.': '1631-0705', - 'can. j. electr. comput. eng.': '0840-8688', - 'cas. pest. mat.': '0862-7959', - 'catal. lett.': '1572-879X', - 'celest. mech. dyn. astron.': '1572-9478', - 'chem. mater.': '1520-5002', - 'chem. vap. depos.': '1521-3862', - 'chemphyschem': '1439-7641', - 'chin. astron.': '0146-6364', - 'chin. librariansh.': '1089-4667', - 'chin. opt. lett.': '1671-7694', - 'chin. phys.': '1741-4199', - 'chin. phys. lett.': '0256-307X', - 'circuit world': '0305-6120', - 'circuits syst. signal process.': '1531-5878', - 'clean technol. environ. policy': '1618-9558', - 'clefs cea': '0298-6248', - 'clin. phys. physiol. meas.': '0143-0815', - 'cluster comput.': '1386-7857', - 'coastal eng.': '0378-3839', - 'colloid j.': '1608-3067', - 'colloq. math.': '1730-6310', - 'comments mod. phys.': '1560-5892', - 'commun. acm': '0001-0782', - 'commun. nonlinear sci. numer. simul.': '1007-5704', - 'commun. pure appl. math.': '1097-0312', - 'commun. soc.: dig. news events interest commun. eng.': '0094-5579', - 'complexity': '1099-0526', - 'compos. math.': '0010-437X', - 'composites': '0010-4361', - 'comput. archit. news': '0163-5964', - 'comput. bull.': '0010-4531', - 'comput. chem. eng.': '0098-1354', - 'comput. commun. rev.': '0146-4833', - 'comput. graph.': '0097-8930', - 'comput. humanit.': '1572-8412', - 'comput. ind. eng.': '0360-8352', - 'comput. integr. manuf. syst.': '0951-5240', - 'comput. math. model.': '1046-283X', - 'comput. math. organ. theory': '1381-298X', - 'comput. netw. isdn syst.': '0169-7552', - 'comput. optim. appl.': '0926-6003', - 'comput. phys. rep.': '0167-7977', - 'comput. soc.': '0095-2737', - 'comput. softw.': '0289-6540', - 'comput. speech lang.': '0885-2308', - 'comput. support. coop. work': '0925-9724', - 'comput. vis. image underst.': '1077-3142', - 'computer': '0018-9162', - 'concurr. comput.: pract. exp.': '1532-0634', - 'concurr.: pract. exp.': '1096-9128', - 'constraints': '1572-9354', - 'contact context': '1547-8890', - 'contrib. plasma phys.': '1521-3986', - 'cosm. res.': '0010-9525', - 'cost eng. j.': '0274-9626', - 'cryst. growth des.': '1528-7483', - 'cryst. res. technol.': '1521-4079', - 'cultiv. interact.': '1471-3225', - 'curr. appl. phys.': '1567-1739', - 'curr. opin. colloid. interface sci.': '1359-0294', - 'cybermetrics': '1137-5019', - 'cybern. syst. anal.': '1060-0396', - 'czechoslov. j. phys.': '1572-9486', - 'czechoslov. math. j.': '0011-4642', - 'data base adv. inf. syst.': '0095-0033', - 'data min. knowl. discov.': '1384-5810', - 'data sci. j.': '1683-1470', - 'des. autom. embed. syst.': '1572-8080', - 'des. codes cryptogr.': '0925-1022', - 'des. monomers polym.': '1568-5551', - 'differ. equ.': '0012-2116', - 'differ. geom.': '1454-511X', - 'digit. signal process.': '1051-2004', - 'discrete event dyn. syst.': '1573-7594', - 'distrib. parallel databases': '0926-8272', - 'documentaliste': '0012-4508', - 'dokl. phys. chem.': '0012-5016', - 'dyn. control': '1573-8450', - 'e-polymers': '1618-7229', - 'e-streams': '1098-4399', - 'earth moon planets': '0167-9295', - 'ec compet. policy newsl.': '1025-2266', - 'educ. inf. technol.': '1360-2357', - 'educ. stud. math.': '1573-0816', - 'egypt. j. solids': '1012-5566', - 'electrodepos. surf. treat.': '0300-9416', - 'electron microsc. rev.': '0892-0354', - 'electron. j. comb.': '1027-5487', - 'electron. j. theor. phys.': '1729-5254', - 'electron. libr.': '0264-0473', - 'electron. res. announc. am. math. soc.': '1079-6762', - 'electron. trans. artif. intell.': '1403-204X', - 'empir. softw. eng.': '1382-3256', - 'entropy': '1099-4300', - 'environ. qual. manag.': '1520-6483', - 'environmetrics': '1099-095X', - 'epj a direct': '1435-3725', - 'epj c direct': '1435-3725', - 'epj e direct': '1435-3725', - 'eso astrophys. symp.': '1611-6143', - 'ethics inf. technol.': '1572-8439', - 'etri j.': '1225-6463', - 'eur. environ.': '1099-0976', - 'eur. j. solid state inorg. chem.': '0992-4361', - 'eur. trans. electr. power': '1546-3109', - 'eur. union': '1472-3395', - 'eurasip j. wirel. commun. netw.': '1687-1499', - 'exergy': '1164-0235', - 'exp. astron.': '1572-9508', - 'extremes': '1572-915X', - 'fire technol.': '1572-8099', - 'fluid dyn.': '1573-8507', - 'form. methods syst. des.': '1572-8102', - 'forschung': '1522-2357', - 'fortran forum': '1061-7264', - 'fortschr. phys.': '1521-3978', - 'found. phys.': '0015-9018', - 'found. phys. lett.': '0894-9875', - 'free online scholarsh. newsl.': '1535-7848', - 'freepint newsl.': '1460-7239', - 'frontiers (swindon)': '1460-5600', - 'fuel cells': '1615-6854', - 'funct. anal. appl.': '0016-2663', - 'fundam. math.': '1730-6329', - 'fuzzy optim. decis. mak.': '1573-2908', - 'gaz. vide': '1638-802X', - 'gen. relativ. gravit.': '1572-9532', - 'geoinformatica': '1384-6175', - 'germ. res.': '1522-2322', - 'glass ceram.': '1573-8515', - 'gps solut.': '1521-1886', - 'graph. models image process.': '1077-3169', - 'heat recovery syst. chp': '0890-4332', - 'high energy chem.': '1608-3148', - 'high energy density phys.': '1574-1818', - 'high energy phys. nucl. phys. (beijing)': '0254-3052', - 'high temp.': '0018-151X', - 'hit j. sci. eng.': '1565-5008', - 'icarus': '0019-1035', - 'icsti forum': '1018-9580', - 'ieee aerosp. electron. syst. mag.': '0885-8985', - 'ieee ann. hist. comput.': '1058-6180', - 'ieee antennas propag. mag.': '1045-9243', - 'ieee antennas wirel. propag. lett.': '1536-1225', - 'ieee assp mag.': '0740-7467', - 'ieee circuits devices mag.': '8755-3996', - 'ieee circuits syst. mag.': '1531-636X', - 'ieee commun. lett.': '1089-7798', - 'ieee commun. mag.': '0163-6804', - 'ieee comput. appl. power': '0895-0156', - 'ieee comput. graph. appl.': '0272-1716', - 'ieee comput. sci. eng.': '1070-9924', - 'ieee concurr.': '1092-3063', - 'ieee control syst. mag.': '0272-1708', - 'ieee des. test comput.': '0740-7475', - 'ieee distrib. syst. online': '1541-4922', - 'ieee electr. insul. mag.': '0883-7554', - 'ieee electron device lett.': '0741-3106', - 'ieee eng. med. biol. mag.': '0739-5175', - 'ieee expert mag.': '0885-9000', - 'ieee instrum. measur. mag.': '1094-6969', - 'ieee intell. syst.': '1541-1672', - 'ieee intell. syst. appl.': '1094-7167', - 'ieee internet comput.': '1089-7801', - 'ieee j. ocean. eng.': '0364-9059', - 'ieee j. quantum electron.': '0018-9197', - 'ieee j. robot. autom.': '0882-4967', - 'ieee j. sel. areas commun.': '0733-8716', - 'ieee j. sel. top. quantum electron.': '1077-260X', - 'ieee j. solid state circuits': '0018-9200', - 'ieee lcs': '1045-9235', - 'ieee lts': '1055-6877', - 'ieee micro': '0272-1732', - 'ieee microw. guid. wave lett.': '1051-8207', - 'ieee microw. mag.': '1527-3342', - 'ieee microw. wirel. compon. lett.': '1531-1309', - 'ieee multimed.': '1070-986X', - 'ieee netw.': '0890-8044', - 'ieee parallel distrib. technol.: syst. appl.': '1063-6552', - 'ieee pers. commun.': '1070-9916', - 'ieee pervasive comput.': '1536-1268', - 'ieee photonics technol. lett.': '1041-1135', - 'ieee potentials': '0278-6648', - 'ieee power electron. lett.': '1540-7985', - 'ieee power energy mag.': '1540-7977', - 'ieee power eng. rev.': '0272-1724', - 'ieee robot. autom. mag.': '1070-9932', - 'ieee secur. priv. mag.': '1540-7993', - 'ieee sens. j.': '1530-437X', - 'ieee signal process. lett.': '1070-9908', - 'ieee signal process. mag.': '1053-5888', - 'ieee softw.': '0740-7459', - 'ieee spectr.': '0018-9235', - 'ieee technol. soc. mag.': '0278-0097', - 'ieee trans. acoust. speech signal process.': '0096-3518', - 'ieee trans. adv. packag.': '1521-3323', - 'ieee trans. aerosp. electron. syst.': '0018-9251', - 'ieee trans. antennas propag.': '0018-926X', - 'ieee trans. appl. supercond.': '1051-8223', - 'ieee trans. audio': '0096-1620', - 'ieee trans. audio electroacoust.': '0018-9278', - 'ieee trans. audio speech lang. process.': '1558-7916', - 'ieee trans. autom. sci. eng.': '1545-5955', - 'ieee trans. automat. control': '0018-9286', - 'ieee trans. biomed. eng.': '0018-9294', - 'ieee trans. broadcast.': '0018-9316', - 'ieee trans. circuits syst.': '0098-4094', - 'ieee trans. circuits syst. video technol.': '1051-8215', - 'ieee trans. circuits syst., i': '1057-7122', - 'ieee trans. circuits syst., ii': '1057-7130', - 'ieee trans. commun.': '0090-6778', - 'ieee trans. compon. hybrids manuf. technol.': '0148-6411', - 'ieee trans. compon. packag. manuf. technol. a': '1070-9886', - 'ieee trans. compon. packag. manuf. technol. b': '1070-9894', - 'ieee trans. compon. packag. manuf. technol. c': '1083-4400', - 'ieee trans. compon. packag. technol.': '1521-3331', - 'ieee trans. compon. parts': '0097-6601', - 'ieee trans. comput.': '0018-9340', - 'ieee trans. comput.-aided des. integrat. circuits syst.': '0278-0070', - 'ieee trans. consum. electron.': '0098-3063', - 'ieee trans. control syst. technol.': '1063-6536', - 'ieee trans. dependable secur. comput.': '1545-5971', - 'ieee trans. device mater. reliab.': '1530-4388', - 'ieee trans. dielectr. electr. insul.': '1070-9878', - 'ieee trans. educ.': '0018-9359', - 'ieee trans. electr. insul.': '0018-9367', - 'ieee trans. electromagn. compat.': '0018-9375', - 'ieee trans. electron devices': '0018-9383', - 'ieee trans. electron. packag. manuf.': '1521-334X', - 'ieee trans. energy convers.': '0885-8969', - 'ieee trans. eng. manag.': '0018-9391', - 'ieee trans. evol. comput.': '1089-778X', - 'ieee trans. fuzzy syst.': '1063-6706', - 'ieee trans. geosci. remote sens.': '0196-2892', - 'ieee trans. image process.': '1057-7149', - 'ieee trans. ind. appl.': '0093-9994', - 'ieee trans. ind. electron.': '0278-0046', - 'ieee trans. ind. inform.': '1551-3203', - 'ieee trans. inf. technol. biomed.': '1089-7771', - 'ieee trans. inf. theory': '0018-9448', - 'ieee trans. instrum. meas.': '0018-9456', - 'ieee trans. intell. transp. syst.': '1524-9050', - 'ieee trans. knowl. data eng.': '1041-4347', - 'ieee trans. magn.': '0018-9464', - 'ieee trans. manuf. technol.': '0046-838X', - 'ieee trans. med. imaging': '0278-0062', - 'ieee trans. microw. theory tech.': '0018-9480', - 'ieee trans. mob. comput.': '1536-1233', - 'ieee trans. multimed.': '1520-9210', - 'ieee trans. nanobiosci.': '1536-1241', - 'ieee trans. nanotechnol.': '1536-125X', - 'ieee trans. neural netw.': '1045-9227', - 'ieee trans. neural syst. rehabil. eng.': '1534-4320', - 'ieee trans. nucl. sci.': '0018-9499', - 'ieee trans. parallel distrib. syst.': '1045-9219', - 'ieee trans. parts hybrids packag.': '0361-1000', - 'ieee trans. parts mater. packag.': '0018-9502', - 'ieee trans. pattern anal. mach. intell.': '0162-8828', - 'ieee trans. plasma sci.': '0093-3813', - 'ieee trans. power deliv.': '0885-8977', - 'ieee trans. power electron.': '0885-8993', - 'ieee trans. power syst.': '0885-8950', - 'ieee trans. prod. eng. prod.': '0097-4544', - 'ieee trans. prof. commun.': '0361-1434', - 'ieee trans. rehabil. eng.': '1063-6528', - 'ieee trans. reliab.': '0018-9529', - 'ieee trans. robot.': '1552-3098', - 'ieee trans. robot. autom.': '1042-296X', - 'ieee trans. semicond. manuf.': '0894-6507', - 'ieee trans. signal process.': '1053-587X', - 'ieee trans. softw. eng.': '0098-5589', - 'ieee trans. sonics ultrason.': '0018-9537', - 'ieee trans. speech audio process.': '1063-6676', - 'ieee trans. syst. man cybern.': '0018-9472', - 'ieee trans. syst. man cybern. a': '1083-4427', - 'ieee trans. syst. man cybern. b': '1083-4419', - 'ieee trans. syst. man cybern. c': '1094-6977', - 'ieee trans. ultrason. eng.': '0893-6706', - 'ieee trans. ultrason., ferroelectr. freq. control': '0885-3010', - 'ieee trans. veh. technol.': '0018-9545', - 'ieee trans. very large scale integr. (vlsi) syst.': '1063-8210', - 'ieee trans. vis. comput. graph.': '1077-2626', - 'ieee trans. wirel. commun.': '1536-1276', - 'ieee wirel. commun.': '1536-1284', - 'ieee/acm trans. netw.': '1063-6692', - 'ieee/asme trans. mechatron.': '1083-4435', - 'iii-vs rev.': '0961-1290', - 'inf. bull. var. stars': '1587-2440', - 'inf. manag.': '0378-7206', - 'inf. organ.': '1471-7727', - 'inf. process. manag.': '0306-4573', - 'inf. res.': '1368-1613', - 'inf. retr.': '1386-4564', - 'inf. sci. appl.': '1069-0115', - 'inf. syst. e-bus. manag.': '1617-9854', - 'inf. syst. front.': '1387-3326', - 'inf. technol. disabil.': '1073-5727', - 'inf. technol. manag.': '1385-951X', - 'infeuro': '1027-930X', - 'infrared phys.': '0020-0891', - 'innov. syst. softw. eng.': '1614-5054', - 'innov. teach. learn. inf. comput. sci.': '1473-1707', - 'innov. technol. transf.': '1013-6452', - 'innov. transf. technol.': '1025-692X', - 'inorg. mater.': '1608-3172', - 'instrum. exp. tech.': '0020-4412', - 'int. appl. mech.': '1573-8582', - 'int. insolv. rev.': '1099-1107', - 'int. j. appl. electromagn. mech.': '1383-5416', - 'int. j. appl. math. comput. sci.': '1641-876X', - 'int. j. appl. radiat. isot.': '0020-708X', - 'int. j. comput. math. learn.': '1382-3892', - 'int. j. comput. vis.': '0920-9429', - 'int. j. des. comput.': '1329-7147', - 'int. j. electron. commun.': '1434-8411', - 'int. j. electron. commun. (aeu)': '1434-8411', - 'int. j. fract.': '0376-9429', - 'int. j. hum.-comput. stud.': '1071-5819', - 'int. j. infrared millim. waves': '1572-9559', - 'int. j. intell. syst.': '1098-111X', - 'int. j. mass spectrom.': '1387-3806', - 'int. j. mass spectrom. ion process.': '0168-1176', - 'int. j. mod. phys. d': '0218-2718', - 'int. j. mod. phys. e': '0218-3013', - 'int. j. parallel program.': '0885-7458', - 'int. j. pattern recognit. artif. intell.': '0218-0014', - 'int. j. prod. econ.': '0925-5273', - 'int. j. radiat. appl. instrum. a': '0883-2889', - 'int. j. radiat. appl. instrum. d': '1359-0189', - 'int. j. radiat. phys. chem. (1969-76)': '0020-7055', - 'int. j. radiat. phys. chem., c': '1359-0197', - 'int. j. rock mech. min. sci.': '1365-1609', - 'int. j. technol. des. educ.': '0957-7572', - 'int. j. theor. phys.': '1572-9575', - 'int. j. therm. sci.': '1290-0729', - 'int. j. thermophys.': '1572-9567', - 'int. j. wirel. inf. netw.': '1068-9605', - 'intel. artif.': '1137-3601', - 'interact. multimed. electron. j. comput. enhanc. learn.': '1525-9102', - 'interface sci.': '0927-7056', - 'ipn sci.': '1622-5120', - 'ire trans. audio': '0096-1981', - 'ire trans. autom. control': '0096-199X', - 'ire trans. circuit theory': '0098-4094', - 'ire trans. compon. parts': '0096-2422', - 'ire trans. prod. eng. prod.': '0096-1779', - 'ire trans. prod. tech.': '0096-1760', - 'ire trans. ultrason. eng.': '0096-1019', - 'it archit.': '1557-2145', - 'it prof.': '1520-9202', - 'itbm-rbm': '1297-9562', - 'itbm-rbm news': '1297-9570', - 'itnow': '1746-5702', - 'j. acm assoc. comput. mach.': '0004-5411', - 'j. adhes. sci. technol.': '1568-5616', - 'j. algebr. comb.': '0925-9899', - 'j. am. soc. inf. sci.': '1097-4571', - 'j. am. soc. inf. sci. technol.': '1532-2890', - 'j. anal. chem.': '1608-3199', - 'j. appl. clin. med. phys.': '1526-9914', - 'j. appl. electrochem.': '0021-891X', - 'j. appl. mech. tech. phys.': '1573-8620', - 'j. appl. spectrosc.': '1573-8647', - 'j. artif. intell. res.': '1076-9757', - 'j. astrophys. astron.': '0250-6335', - 'j. autom. reason.': '0168-7433', - 'j. biomater. sci., polym. ed.': '1568-5624', - 'j. braz. comput. soc.': '0104-6500', - 'j. chem. doc.': '1961-1974', - 'j. chem. eng. data': '1520-5134', - 'j. chemom.': '1099-128X', - 'j. colloid interface sci.': '0021-9797', - 'j. comput. aided mater. des.': '0928-1045', - 'j. comput. anal. appl.': '1521-1398', - 'j. comput. electron.': '1569-8025', - 'j. comput. neurosci.': '0929-5313', - 'j. comput. phys.': '0021-9991', - 'j. comput. sci. technol.': '1860-4749', - 'j. comput.- mediat. commun.': '1083-6101', - 'j. corros. sci. eng.': '1466-8858', - 'j. cosmol. astropart. phys.': '1475-7516', - 'j. data sci.': '1683-8602', - 'j. des. commun.': '1137-3601', - 'j. digit. inf.': '1368-7506', - 'j. disp. technol.': '1551-319X', - 'j. dyn. control syst.': '1079-2724', - 'j. dyn. differ. equ.': '1040-7294', - 'j. elast.': '1573-2681', - 'j. electroceram.': '1385-3449', - 'j. electromagn. waves appl.': '1569-3937', - 'j. electron. test.': '0923-8174', - 'j. eng. math.': '0022-0833', - 'j. eng. phys. thermophys.': '1573-871X', - 'j. fluids struct.': '0889-9746', - 'j. fourier anal. appl.': '1531-5851', - 'j. fusion energy': '1572-9591', - 'j. geophys. eng.': '1742-2132', - 'j. glob. optim.': '0925-5001', - 'j. grid comput.': '1572-9814', - 'j. heuristics': '1381-1231', - 'j. high energy phys.': '1126-6708', - 'j. instrum.': '1748-0221', - 'j. intell. inf. syst.': '0925-9902', - 'j. intell. manuf.': '1572-8145', - 'j. intell. robot. syst.': '1573-0409', - 'j. interlibr. loan doc. deliv. electron. reserve': '1072-303X', - 'j. jpn. stat. soc.': '1348-6365', - 'j. lightwave technol.': '0733-8724', - 'j. log. algebr. program.': '1567-8326', - 'j. log. lang. inf.': '1572-9583', - 'j. low temp. phys.': '1573-7357', - 'j. magn. reson.': '1090-7807', - 'j. magn. reson. a': '1064-1858', - 'j. magn. reson. b': '1064-1866', - 'j. magn. reson. imag.': '1522-2586', - 'j. mater. cycles waste manag.': '1611-8227', - 'j. mater. sci.': '0022-2461', - 'j. mater. sci. lett.': '0261-8028', - 'j. mater. sci.: mater. electron.': '0957-4522', - 'j. mater. sci.: mater. med.': '0957-4530', - 'j. mater. synth. process.': '1573-4870', - 'j. math. imaging vis.': '0924-9907', - 'j. math. model. algorithms': '1570-1166', - 'j. math. sci.': '1072-3374', - 'j. math. teach. educ.': '1573-1820', - 'j. microcomput. appl.': '0745-7138', - 'j. microelectromech. syst.': '1057-7157', - 'j. micromechatron.': '1568-5632', - 'j. nanomater.': '1687-4129', - 'j. nanopart. res.': '1572-896X', - 'j. netw. comput. appl.': '1084-8045', - 'j. netw. syst. manag.': '1064-7570', - 'j. neural eng.': '1741-2560', - 'j. non-newton. fluid mech.': '0377-0257', - 'j. nondestruct. eval.': '0195-9298', - 'j. nucl. energy, c': '0368-3281', - 'j. object technol.': '1660-1769', - 'j. oper. manage.': '0272-6963', - 'j. opt.': '0150-536X', - 'j. opt. fiber commun. rep.': '1619-8638', - 'j. opt. netw.': '1536-5379', - 'j. opt. technol.': '1070-9762', - 'j. optim. theory appl.': '0022-3239', - 'j. parallel distrib. comput.': '0743-7315', - 'j. phys. c': '0022-3719', - 'j. phys. chem. a': '0092-7325', - 'j. phys. chem. b': '0092-7325', - 'j. phys. f': '0305-4608', - 'j. phys. stud.': '1027-4642', - 'j. phys.: conf. ser.': '1742-6596', - 'j. polym. res.': '1022-9760', - 'j. porous mater.': '1573-4854', - 'j. product. anal.': '1573-0441', - 'j. radiat. res.': '0449-3060', - 'j. radioanal. nucl. chem.': '1588-2780', - 'j. res. natl. inst. stand. technol.': '1044-677X', - 'j. res. pract.': '1712-851X', - 'j. russ. laser res.': '1573-8760', - 'j. sci. commun.': '1824-2049', - 'j. sci. comput.': '0885-7474', - 'j. sci. instrum.': '0950-7671', - 'j. soc. radiol. prot.': '0260-2814', - 'j. softw. maint. evol.: res. pract.': '1532-0618', - 'j. softw. maint.: res. pract.': '1096-908X', - 'j. sound vib.': '0022-460X', - 'j. south. acad. spec. librariansh.': '1525-321X', - 'j. stat. mech., theory exp.': '1742-5468', - 'j. stat. phys.': '1572-9613', - 'j. stat. softw.': '1548-7660', - 'j. strain anal. eng. des.': '0309-3247', - 'j. supercomput.': '0920-8542', - 'j. supercond.': '1572-9605', - 'j. supercond. novel magn.': '1557-1939', - 'j. supercrit. fluids': '0896-8446', - 'j. syst. integr.': '1573-8787', - 'j. technol. transf.': '0829-9912', - 'j. theor. probab.': '0894-9840', - 'j. therm. anal. calorim.': '1572-8943', - 'j. vis. commun. image represent.': '1047-3203', - 'j. vis. comput. animat.': '1099-1778', - 'j. vlsi signal process. syst. signal image video technol.': '0922-5773', - 'jpn. j. appl. phys.': '1347-4065', - 'k-theory': '1573-0514', - 'katharine sharp rev.': '1083-5261', - 'kek news': '1343-3547', - 'lasers med. sci.': '1435-604X', - 'lett. math. phys.': '1573-0530', - 'libr. philos. pract.': '1522-0222', - 'linux j.': '1075-3583', - 'lith. math. j.': '0363-1672', - 'living rev. sol. phys.': '1614-4961', - 'low temp. phys.': '1063-777X', - 'mach. learn.': '0885-6125', - 'macromol. chem. phys.': '1521-3935', - 'macromol. mater. eng.': '1439-2054', - 'macromol. rapid commun.': '1521-3927', - 'macromol. symp.': '1521-3900', - 'macromol. theory simul.': '1521-3919', - 'magma magn. reson. mater. phys. biol. med.': '1352-8661', - 'magn. reson. imaging': '0730-725X', - 'mater. sci.': '1068-820X', - 'mater. technol.': '1580-3414', - 'math. notes': '0001-4346', - 'math. phys. anal. geom.': '1385-0172', - 'math. probl. eng.': '1563-5147', - 'math. scand.': '0025-5521', - 'mc j.': '1069-6792', - 'meas. tech.': '0543-1972', - 'meccanica': '1572-9648', - 'mech. compos. mater.': '1573-8922', - 'mech. syst. signal process.': '0888-3270', - 'mech. time-depend. mater.': '1573-2738', - 'med. phys.': '0094-2405', - 'mediterr. j. math.': '1660-5454', - 'met. sci. heat treat.': '0026-0673', - 'metallurgist': '1573-8892', - 'methodol. comput. appl. probab.': '1387-5841', - 'metrika': '1436-5057', - 'metrologia': '1681-7575', - 'microfluid. nanofluid.': '1613-4990', - 'micromater. nanomater.': '1619-2486', - 'micron': '0968-4328', - 'micron (1969-83) [online version]': '0047-7206', - 'micron microsc. acta': '0739-6260', - 'microw. rf': '0745-2993', - 'milan j. math.': '1424-9294', - 'minds mach.': '1572-8641', - 'minerva': '0026-4695', - 'mo. j. math. sci.': '0899-6180', - 'mob. netw. appl.': '1572-8153', - 'mol. eng.': '1572-8951', - 'monogr. mat.': '0077-0507', - 'monte carlo methods appl.': '1569-3961', - 'mrs bull.': '0883-7694', - 'multibody syst. dyn.': '1573-272X', - 'multidimens. syst. signal process.': '0923-6082', - 'multimed. tools appl.': '1380-7501', - 'm\xc3\xa9c. ind.': '1296-2139', - 'nagoya math. j.': '0027-7630', - 'netw. comput.': '1046-4468', - 'networks': '1097-0037', - 'neural process. lett.': '1370-4621', - 'neutron news': '1044-8632', - 'new dir. high. educ.': '1522-2322', - 'new dir. instit. res.': '1536-075X', - 'new dir. stud. serv.': '1536-0695', - 'new dir. teach. learn.': '1536-0768', - 'nexus netw. j.': '1522-4600', - 'nonlinear dyn.': '1573-269X', - 'nonlinear phenom. complex syst.': '1561-4085', - 'nonprofit couns.': '1520-6785', - 'not. am. math. soc.': '1088-9477', - 'nouv. rev. opt.': '0335-7368', - 'nouv. rev. opt. appl.': '0029-4780', - 'ntm int. j. hist. ethics nat. sci. technol. med.': '1420-9144', - 'nucl. data sheets': '0090-3752', - 'nucl. data sheets, a': '0550-306X', - 'nucl. data sheets, b': '0090-550X', - 'nucl. eng. des. fusion': '0167-899X', - 'nucl. eng. technol.': '1738-5733', - 'nucl. fusion': '0029-5515', - 'nucl. instrum.': '0369-643X', - 'nucl. instrum. methods': '0029-554X', - 'nucl. instrum. methods phys. res.': '0167-5087', - 'nucl. instrum. methods phys. res., a': '0167-5087', - 'nucl. phys.': '0029-5582', - 'nucl. phys. news': '1050-6896', - 'nucl. struct. eng.': '0369-5816', - 'nucl. track detect.': '0145-224X', - 'nucl. tracks': '0735-245X', - 'nucl. tracks radiat. meas.': '0191-278X', - 'nucl. tracks radiat. meas. (1982-85)': '0735-245X', - 'nucl. tracks radiat. meas. (1993)': '0969-8078', - 'nukleonika': '1508-5791', - 'numer. algorithms': '1572-9265', - 'numer. methods partial differ. equ.': '1098-2426', - 'nuovo cimento, riv.': '0393-697X', - 'ocean dyn.': '1616-7228', - 'open syst. inf. dyn.': '1230-1612', - 'oper. syst. rev.': '0163-5980', - 'opt. fiber technol.': '1068-5200', - 'opt. netw. mag.': '1572-8161', - 'opt. photonics news': '1047-6938', - 'opt. quantum electron.': '0306-8919', - 'opt. rev.': '1349-9432', - 'opt. spectrosc.': '1562-6911', - 'opt. switch. netw.': '1573-4277', - 'opt. technol.': '0374-3926', - 'optik': '0030-4026', - 'optim. control appl. methods': '1099-1514', - 'optim. eng.': '1389-4420', - 'oxid. met.': '0030-770X', - 'packag. technol. sci.': '1099-1522', - 'pamm': '1617-7061', - 'part. part. syst. charact.': '1521-4117', - 'period. math. hung.': '1588-2829', - 'pers. technol.': '1433-3066', - 'pers. ubiquitous comput.': '1617-4917', - 'philips j. res.': '0165-5817', - 'photonic netw. commun.': '1572-8188', - 'photonics nanostruct., fundam. appl.': '1569-4410', - 'phys. biol.': '1478-3975', - 'phys. earth planet. inter.': '0031-9201', - 'phys. fluids (1958-88)': '0031-9171', - 'phys. j.': '0031-9279', - 'phys. j. indones. phys. soc.': '1410-8860', - 'phys. lett.': '0031-9163', - 'phys. lett., a': '0375-9601', - 'phys. lett., b': '0370-2693', - 'phys. life rev.': '1571-0645', - 'phys. rev.': '0031-899X', - 'phys. rev. (ser. i)': '0031-899X', - 'phys. rev. lett.': '0031-9007', - 'phys. rev. spec. top. phys. educ. res.': '1554-9178', - 'phys. rev., a': '1050-2947;', - 'phys. rev., b': '0163-1829', - 'phys. rev., c': '0556-2813', - 'phys. rev., d': '0556-2821', - 'phys. rev., e': '1063-651x', - 'phys. status solidi, c': '1610-1642', - 'phys. technol.': '0305-4624', - 'phys. unserer zeit': '1521-3943', - 'physica': '0031-8914', - 'physica b c': '0378-4363', - 'plasma chem. plasma process.': '1572-8986', - 'plasma phys.': '0032-1028', - 'plasmas ions': '1288-3255', - 'plasmas polym.': '1572-8978', - 'poiesis prax.': '1615-6617', - 'polym. gels netw.': '0966-7822', - 'powder metall. met. ceram.': '1068-1302', - 'power technol. eng.': '1570-1468', - 'prace mat.- fiz.': '0867-5570', - 'probab. surv.': '1549-5787', - 'probl. inf. transm.': '0032-9460', - 'proc. ieee': '0018-9219', - 'proc. indian acad. sci., math. sci.': '0253-4142', - 'proc. jpn. acad. a': '0386-2194', - 'proc. phys. soc. (1926-48)': '0959-5309', - 'proc. phys. soc. (1958-67)': '0370-1328', - 'proc. phys. soc. lond.': '1478-7814', - 'proc. phys. soc., a': '0370-1298', - 'proc. phys. soc., b': '0370-1301', - 'prog. cryst. growth charact.': '0146-3535', - 'prog. nucl. magn. reson. spectrosc.': '0079-6565', - 'prog. phys.': '1555-5615', - 'prog. theor. phys., suppl.': '0375-9687', - 'progr. struct. eng. mater.': '1528-2716', - 'program. comput. softw.': '0361-7688', - 'propellants explos. pyrotech.': '1521-4087', - 'prot. met.': '0033-1732', - 'publ. math. ihes': '1618-1913', - 'public underst. sci.': '1361-6609', - 'pure appl. opt.': '0963-9659', - 'qual. assur. j.': '1099-1786', - 'qual. reliab. eng. int.': '1099-1638', - 'quant. financ.': '1469-7688', - 'quantum inf. process.': '1570-0755', - 'quantum opt.': '0954-8998', - 'quantum semiclass. opt.': '1355-5111', - 'queueing syst.': '1572-9443', - 'radiat. phys. chem. (1977-85)': '0146-5724', - 'radiochemistry': '1608-3288', - 'radioisotopes': '0033-8303', - 'radiophys. quantum electron.': '1573-9120', - 'radioprotection': '1769-700X', - 'ramanujan j.': '1572-9303', - 'rbm-news': '0222-0776', - 'real time imaging': '1077-2014', - 'real time syst.': '0922-6443', - 'refract. ind. ceram.': '1083-4877', - 'reliab. comput.': '1385-3139', - 'rend. lincei': '1720-0768', - 'rend. lincei sci. fis. nat.': '1720-0776', - 'rend. semin. mat.': '0373-1243', - 'res. inf.': '1744-8026', - 'res. lang. comput.': '1572-8706', - 'res. nondestruct. eval.': '1432-2110', - 'rev. electron. suisse sci. inf.': '1661-1802', - 'rev. g\xc3\xa9n. therm.': '0035-3159', - 'rev. mex. fis.': '0035-001X', - 'rev. phys. chim. appl. ind.': '1153-9771', - 'rheol. acta': '1435-1528', - 'risonyt': '0108-0350', - 'rom. rep. phys.': '1221-1451', - 'rozpr. mat.': '0860-2581', - 'russ. j. electrochem.': '1023-1935', - 'russ. j. nondestruct. test.': '1061-8309', - 'russ. j. numer. anal. math. model.': '1569-3988', - 'russ. microelectron.': '1063-7397', - 'russ. phys. j.': '1573-9228', - 's. afr. j. inf. manag.': '1560-683X', - 'sankhya. indian j. stat.': '0036-4452', - 'sci. am.': '0036-8733', - 'sci. avenir': '0036-8636', - 'sci. educ.': '1098-237X', - 'sci. soils': '1432-9492', - 'sci. vie hors s\xc3\xa9r.': '0151-0282', - 'scientometrics': '0138-9130', - 'semicond. int.': '0163-3767', - 'semicond. phys. quantum electron. optoelectron.': '1605-6582', - 'semigroup forum': '0037-1912', - 'sens. actuators a': '0924-4247', - 'sens. actuators b': '0925-4005', - 'sens. update': '1616-8984', - 'sensors': '1424-8220', - 'ser. enews': '1476-0576', - 'serials': '0953-0460', - 'sib. math. j.': '0037-4466', - 'sigact news': '0163-5700', - 'sigbio newsl.': '0163-5697', - 'sigcse bull.': '0097-8418', - 'sigsam bull.': '0163-5824', - 'simul. model. pract. theory': '1569-190X', - 'single mol.': '1438-5171', - 'softw. eng. notes': '0163-5948', - 'softw. focus': '1529-7950', - 'softw. process: improv. pract.': '1099-1670', - 'softw. qual. j.': '0963-9314', - 'softw. syst. model.': '1619-1374', - 'softw. test. verif. reliab.': '1099-1689', - 'softw.- pract. exp.': '1097-024X', - 'sol. syst. res.': '0038-0946', - 'solaris': '1265-4876', - 'solid state sci.': '1293-2558', - 'space sci. rev.': '0038-6308', - 'sparc open access newsl.': '1546-7821', - 'stat. comput.': '0960-3174', - 'stat. methods appl.': '1613-981X', - 'stat. sci.': '0883-4237', - 'stnews': '1040-1229', - 'strength mater.': '1573-9325', - 'struct. des. tall build.': '1099-1794', - 'stud. hist. philos. mod. phys.': '1355-2198', - 'studia log.': '1572-8730', - 'studia math.': '1730-6337', - 'subsurf. sens. technol. appl.': '1573-9317', - 'superlattices microstruct.': '0749-6036', - 'surf. sci. lett.': '0167-2584', - 'surf. technol.': '0376-4583', - 'surv. high energy phys.': '0142-2413', - 'synthese': '1573-0964', - 'syst. comput. jpn.': '1520-684X', - 'syst. eng.': '1520-6858', - 'taiwan. j. math.': '1027-5487', - 'telecommun. syst.': '1018-4864', - 'theor. math. phys.': '1573-9333', - 'theory comput.': '1557-2862', - 'theory pract. object syst.': '1096-9942', - 'trans. ire prof. group commun. syst.': '0277-6243', - 'trans. ire prof. group compon. parts': '0096-2422', - 'trans. ire prof. group ultrason. eng.': '0277-626X', - 'trans. jpn. soc. artif. intell.': '1346-8030', - 'trans. opt. soc.': '1475-4878', - 'tribol. lett.': '1573-2711', - 'tsinghua sci. technol.': '1007-0214', - 'turk. j. math.': '1300-0098', - 'turk. j. phys.': '1300-0101', - 'ukr. math. j.': '0041-5995', - 'ultrason. imaging': '0161-7346', - 'univers. access inf. soc.': '1615-5297', - 'upgrade': '1684-5285', - 'user model. user adapt. interact.': '1573-1391', - 'uspekhi fiz. nauk': '0042-1294', - 'vak. forsch. prax.': '1522-2454', - 'vine': '1474-1032', - 'virtual real.': '1434-9957', - 'web semant.': '1570-8268', - 'weld. int.': '1573-9449', - 'wind energy': '1099-1824', - 'wirel. commun. mob. comput.': '1530-8677', - 'wirel. netw.': '1022-0038', - 'wirel. pers. commun.': '0929-6212', - 'world pat. inf.': '0172-2190', - 'world wide web': '1386-145X', - 'z. anal. anwend.': '0232-2064', - 'z. angew. math. mech.': '1521-4001', - 'z. krist.gr.': '0044-2968', - 'z. phys.': '0044-3328', - 'z. phys., c': '0170-9739'} - -def format_element(bfo): - """ - Returns the ISSN of the record, if known.
    - Note that you HAVE to pre-generate the correspondances - journal->ISSN if you want this element - to return something (Run python bfe_issn.py -h to get help). - """ - - journal_name = bfo.field(journal_name_tag) - - # Here you might want to process journal name - # by doing the same operation that has been - # done when saving the mappings - journal_name = journal_name.lower().strip() - if journal_name.endswith("[online]"): - journal_name = journal_name[:-8].rstrip() - - return issns.get(journal_name, '') - -def build_issns_from_distant_site(url): - """ - Retrieves the ISSNs from a distant Invenio system. - Store the "journal name -> issn" relation. - - Normalize journal names a little bit: - - strip whithespace chars (left and right) - - all lower case - - remove "[Online]" suffix - - Print the result as Python dict structure. - - @param url: the url to load issn from (in the *exact* form: http://www.mysite.com/) - """ - ## Parse the results of the http request: - ## http://cds.cern.ch/search?cc=Periodicals&ot=022,210&of=tm&rg=9000 - - pattern_field = re.compile(r''' - \D*(?P\d*) #document id - \s(?P\d*)__\s\$\$a #tag - (?P.*?)$ #value - ''', re.IGNORECASE | re.DOTALL | re.VERBOSE) - request = '/search?cc=Periodicals&ot=022,210&of=tm' - try: - fields = BIBFORMAT_OPENER.open(url.rstrip('/') + request).readlines() - except IOError: - sys.stderr.write("Error: Could not connect to %s.\n" % url) - sys.exit(0) - - last_doc_id = None - last_issn = None - - built_issns = {} - #built_issns = issns # Uncomment this to extend existing issns dict - # (e.g. in case of manual addition) - for field in fields: - result = pattern_field.search(field) - if result: - doc_id = result.group('docid') - if doc_id != last_doc_id: - # Reset saved ISSN if we parse new document - last_issn = None - - tag = result.group('tag') - if tag == '022': - # Remember this ISSN - last_issn = result.group('value') - - elif tag == '210' and last_issn is not None: - # Found a journal name and issn exists. - # Depending on how journal names are entered into the - # database, you might want to do some processing - # before saving: - journal = result.group('value') - journal = journal.lower().strip() - if journal.endswith("[online]"): - journal = journal[:-8].rstrip() - - built_issns[journal] = last_issn - - last_doc_id = doc_id - - prtyp = pprint.PrettyPrinter(indent=4) - prtyp.pprint(built_issns) - -def build_issns_from_local_site(): - """ - Retrieves the ISSNs from the local database. - Store the "journal name -> issn" relation. - - Normalize journal names a little bit: - - strip whithespace chars (left and right) - - all lower case - - remove "[Online]" suffix - - Print the result as Python dict structure. - """ - - rec_id_list = perform_request_search(cc='Periodicals', - of='id') - built_issns = {} - #built_issns = issns # Uncomment this to extend existing issns dict - # (e.g. in case of manual addition) - for rec_id in rec_id_list: - journal_name_list = get_fieldvalues(rec_id, '210__%') - issn_list = get_fieldvalues(rec_id, '022__a') - if issn_list: - issn = issn_list[0] # There should be only one ISSN - for journal_name in journal_name_list: - # Depending on how journal names are entered into the database, - # you might want to do some processing before saving: - journal_name = journal_name.lower().strip() - if journal_name.endswith("[online]"): - journal_name = journal_name[:-8].rstrip() - - built_issns[journal_name] = issn - - prtyp = pprint.PrettyPrinter(indent=4) - prtyp.pprint(built_issns) - -def print_info(): - """ - Info on element arguments - """ - print(""" Collects ISSN and corresponding journal names from local repository - and prints archive as dict structure. - - Usage: python bfe_issn.py [Options] [url] - Example: python bew_issn.py http://cds.cern.ch/ - - Options: - -h, --help print this help - -u, --url the URL to collect ISSN from - -v, --version print version number - - If 'url' is not given, collect from local database, using a faster method. - - Returned structure can then be copied into bfe_issn.py - 'format' function. - """) -if __name__ == '__main__': - try: - opts, args = getopt.getopt(sys.argv[1:], "hu:v", - ["help", - "url", - "version" - ]) - except getopt.error: - print_info() - sys.exit(0) - - url_arg = None - for opt, opt_value in opts: - - if opt in ["-u", "--url"]: - url_arg = opt_value - elif opt in ["-v", "--version"]: - print(__revision__) - sys.exit(0) - else: - print_info() - sys.exit(0) - - if url_arg is not None: - build_issns_from_distant_site(url_arg) - else: - build_issns_from_local_site() diff --git a/invenio/modules/formatter/format_elements/bfe_keywords.py b/invenio/modules/formatter/format_elements/bfe_keywords.py deleted file mode 100644 index 8f849dbed..000000000 --- a/invenio/modules/formatter/format_elements/bfe_keywords.py +++ /dev/null @@ -1,62 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints keywords -""" -__revision__ = "$Id$" - -import cgi -from urllib import quote -from invenio.config import CFG_BASE_URL - - -def format_element(bfo, keyword_prefix, keyword_suffix, separator=' ; ', - link='yes'): - """ - Display keywords of the record. - - @param keyword_prefix: a prefix before each keyword - @param keyword_suffix: a suffix after each keyword - @param separator: a separator between keywords - @param link: links the keywords if 'yes' (HTML links) - """ - keywords = bfo.fields('6531_a') - - if len(keywords) > 0: - if link == 'yes': - keywords = ['' + cgi.escape(keyword) + '' - for keyword in keywords] - else: - keywords = [cgi.escape(keyword) - for keyword in keywords] - - keywords = [keyword_prefix + keyword + keyword_suffix - for keyword in keywords] - return separator.join(keywords) - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_meta.py b/invenio/modules/formatter/format_elements/bfe_meta.py deleted file mode 100644 index 4847df681..000000000 --- a/invenio/modules/formatter/format_elements/bfe_meta.py +++ /dev/null @@ -1,247 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2013, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - meta""" - -__revision__ = "$Id$" - -import cgi -import re -from invenio.utils.date import strftime, strptime -import string -from datetime import datetime -from invenio.modules.formatter.format_elements.bfe_server_info import format_element as server_info -from invenio.modules.formatter.format_elements.bfe_client_info import format_element as client_info -from invenio.utils.date import get_i18n_month_name -from invenio.utils.html import create_tag -from invenio.legacy.bibindex.engine_utils import get_field_tags -from invenio.config import \ - CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR, \ - CFG_WEBSEARCH_ENABLE_OPENGRAPH, \ - CFG_SITE_LANG, \ - CFG_CERN_SITE - -def format_element(bfo, name, tag_name='', tag='', kb='', kb_default_output='', var='', protocol='googlescholar'): - """Prints a custom field in a way suitable to be used in HTML META - tags. In particular conforms to Google Scholar harvesting protocol as - defined http://scholar.google.com/intl/en/scholar/inclusion.html and - Open Graph http://ogp.me/ - - @param tag_name: the name, from tag table, of the field to be exported - looks initially for names prefixed by "meta-" - then looks for exact name, then falls through to "tag" - @param tag: the MARC tag to be exported (only if not defined by tag_name). Comma-separated list of tags. - @param name: name to be displayed in the meta headers, labelling this value. - @param kb: a knowledge base through which to process the retrieved value if necessary. - @param kb: when a 'kb' is specified and no match for value is found, what shall we - return? Either return the given parameter or specify "{value}" to return the retrieved - value before processing though kb. - @param var: the name of a variable to output instead of field from metadata. - Allowed values are those supported by bfe_server_info and - bfe_client_info. Overrides name and tag_name - @param protocol: the protocol this tag is aimed at. Can be used to switch on/off support for a given "protocol". Can take values among 'googlescholar', 'opengraph' - @see: bfe_server_info.py, bfe_client_info.py - """ - if protocol == 'googlescholar' and not CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR: - return "" - elif protocol == 'opengraph' and not CFG_WEBSEARCH_ENABLE_OPENGRAPH: - return "" - - matched_by_tag_name_p = False - tags = [] - if var: - # delegate to bfe_server_info or bfe_client_info: - value = server_info(bfo, var) - if value.startswith("Unknown variable: "): - # Oops variable was not defined there - value = client_info(bfo, var) - return not value.startswith("Unknown variable: ") and \ - create_metatag(name=name, content=cgi.escape(value, True)) \ - or "" - elif tag_name: - # First check for special meta named tags - tags = get_field_tags("meta-" + tag_name) - if not tags: - # then check for regular tags - tags = get_field_tags(tag_name) - matched_by_tag_name_p = tags and True or False - if not tags and tag: - # fall back to explicit marc tag - if ',' in tag: - tags = tag.split(',') - else: - tags = [tag] - if not tags: - return '' - out = [] - - if protocol == 'googlescholar' and \ - (tags == ['100__a'] or tags == ['700__a']): - # Authors for Google Scholar: remove names that are not purely - # author (thesis director, coordinator, etc). Assume that - # existence of $e subfield is a sign. Since this assumption - # might be wrong, put some strong conditions in order to get - # into this branch, with easy way to bypass. - values = [field_instance[tags[0][-1]] for field_instance in bfo.fields(tags[0][:-1], escape=9) \ - if not field_instance.has_key('e')] - else: - # Standard fetching of values - values = [bfo.fields(marctag, escape=9) for marctag in tags] - - - if name == 'citation_dissertation_institution': - if CFG_CERN_SITE and \ - 'THESIS' in bfo.fields('980__a'): - authors = bfo.fields('100__', escape=9) - authors.extend(bfo.fields('700__', escape=9)) - values = [field_instance['u'] for field_instance in authors \ - if not field_instance.has_key('e') and field_instance.has_key('u')] - elif tag == '100__u' and not matched_by_tag_name_p: - # TODO: find way to map correctly this tag - values = [] - - for value in values: - if isinstance(value, list): - for val in value: - if isinstance(val, dict): - out.extend(val.values()) - else: - out.append(val) - elif isinstance(value, dict): - out.extend(value.values()) - else: - out.append(value) - - if name == 'citation_date': - for idx in range(len(out)): - out[idx] = out[idx].replace('-', '/') - - elif name == 'citation_publication_date': - for idx in range(len(out)): - # Stop at first match - parsed_date = parse_date_for_googlescholar(out[idx]) - if parsed_date: - out = [parsed_date] - break - - out = dict(zip(out, len(out)*[''])).keys() # Remove duplicates - - if kb: - if kb_default_output == "{value}": - out = [bfo.kb(kb, value, value) for value in out] - else: - out = [bfo.kb(kb, value, kb_default_output) for value in out] - return '\n'.join([create_metatag(name=name, content=value) for value in out]) - -def create_metatag(name, content): - """ - Wraps create_tag - """ - if name.startswith("og:"): - return create_tag('meta', property=name, content=content) - else: - return create_tag('meta', name=name, content=content) - -# Build list of expected patterns -CFG_POSSIBLE_DATE_FORMATS = ['%Y %m %d', - '%d %m %Y', - '%d %B %Y', - '%d %b %Y', - '%Y %B %d', - '%Y %b %d', - '%m %Y', - '%B %Y', - '%b %Y', - '%x', - '%x %X'] -CFG_POSSIBLE_DATE_FORMATS = CFG_POSSIBLE_DATE_FORMATS + \ - [f + ' %H:%M:%S' for f in CFG_POSSIBLE_DATE_FORMATS] + \ - [f + 'T%H:%M:%S' for f in CFG_POSSIBLE_DATE_FORMATS] - -# Build month translation mapping from CFG_SITE_LANG to English. -CFG_MONTH_NAMES_MAPPING = {} -if CFG_SITE_LANG != 'en': - CFG_MONTH_NAMES_MAPPING = dict([(get_i18n_month_name(month_nb, display='short', ln=CFG_SITE_LANG).upper(), - get_i18n_month_name(month_nb, display='short', ln='en')) \ - for month_nb in range(1, 13)]) - CFG_MONTH_NAMES_MAPPING.update(dict([(get_i18n_month_name(month_nb, display='long', ln=CFG_SITE_LANG).upper(), - get_i18n_month_name(month_nb, display='long', ln='en')) \ - for month_nb in range(1, 13)])) -# Build regular expression pattern to match month name in -# CFG_SITE_LANG. Note the use \W instead of \b for word boundaries -# because of a bug in 're' module with unicode. -CFG_MONTHS_I18N_PATTERN_RE = re.compile(r"(\W|\A)(%s)(\W|\Z)" % "|".join(CFG_MONTH_NAMES_MAPPING.keys()), - re.IGNORECASE | re.UNICODE) - -CFG_YEAR_PATTERN_RE = re.compile("((\D|\A)(\d{4})(\D|\Z))") - -CFG_PUNCTUATION_PATTERN_RE = re.compile(r'[' + string.punctuation + ']') -CFG_SPACES_PATTERN_RE = re.compile(r'\s+') - -def parse_date_for_googlescholar(datetime_string): - """ - Parse (guess) and return the date in a format adequate for Google - Scholar. We don't use dateutils.guess_datetime() as this one might - lead to results not accurate enough. - """ - datetime_string = CFG_PUNCTUATION_PATTERN_RE.sub(' ', datetime_string) - datetime_string = CFG_SPACES_PATTERN_RE.sub(' ', datetime_string) - - def replace_month(match_obj): - "Return translated month in the matching object" - month = match_obj.group(2).strip() - return match_obj.group(1) + \ - CFG_MONTH_NAMES_MAPPING.get(month.upper(), month) + \ - match_obj.group(3) - - parsed_datetime = None - for dateformat in CFG_POSSIBLE_DATE_FORMATS: - try: - parsed_datetime = strptime(datetime_string.strip(), dateformat) - break - except: - pass - - if not parsed_datetime: - # Do it all again, with the translated version of the string - translated_datetime_string = CFG_MONTHS_I18N_PATTERN_RE.sub(replace_month, datetime_string) - for dateformat in CFG_POSSIBLE_DATE_FORMATS: - try: - parsed_datetime = strptime(translated_datetime_string.strip(), dateformat) - break - except: - pass - - if parsed_datetime: - return strftime('%Y/%m/%d', parsed_datetime) - else: - # Look for a year inside the string: - try: - return CFG_YEAR_PATTERN_RE.search(datetime_string).group(3) - except: - return '' - - return '' - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_meta_files.py b/invenio/modules/formatter/format_elements/bfe_meta_files.py deleted file mode 100644 index 14fe45b53..000000000 --- a/invenio/modules/formatter/format_elements/bfe_meta_files.py +++ /dev/null @@ -1,78 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - return the files of a record""" - -from invenio.config import CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR -from invenio.modules.formatter.format_elements.bfe_fulltext import get_files -from invenio.legacy.bibdocfile.api import BibRecDocs, decompose_bibdocfile_url - -def format_element(bfo, file_format='pdf'): - """Return the files attached to this record, in order to be - embedded as a Google Scholar tag. - - @param file_format: the format to include in this output - """ - if not CFG_WEBSEARCH_ENABLE_GOOGLESCHOLAR: - return "" - - bibarchive = BibRecDocs(bfo.recID) - - (files, old_versions_p, additionals_p) = get_files(bfo) - filtered_files = [] - - if files.has_key('main_urls') and \ - files['main_urls'].has_key('Main'): - filtered_files = [f[0] for f in files['main_urls']['Main'] if f[2] == file_format and \ - not url_is_hidden(f[0], bibarchive)] - if not filtered_files: - # Fall back to other doctypes - if files.has_key('main_urls'): - for doctype, list_of_files in files['main_urls'].iteritems(): - filtered_files.extend([f[0] for f in list_of_files if f[2] == file_format and \ - not url_is_hidden(f[0], bibarchive)]) - if not filtered_files: - # Fall back to external urls - if files.has_key('others_urls'): - filtered_files.extend([file_url for file_url, file_name in files['others_urls'] \ - if file_url.endswith('.' + file_format)]) - - tags = ['' % url for url in filtered_files] - - return "\n".join(tags) - - -def url_is_hidden(url, bibarchive): - """ - Return True if the given URL should be hidden according to given - BibRecDocs structure. - """ - try: - (recid, docname, docformat) = decompose_bibdocfile_url(url) - return bibarchive.get_bibdoc(docname).hidden_p(docformat) - except: - return False - return False - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_meta_opengraph_image.py b/invenio/modules/formatter/format_elements/bfe_meta_opengraph_image.py deleted file mode 100644 index 2911e5891..000000000 --- a/invenio/modules/formatter/format_elements/bfe_meta_opengraph_image.py +++ /dev/null @@ -1,119 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - return an image for the record""" - -from invenio.config import CFG_SITE_URL, CFG_SITE_SECURE_URL, CFG_CERN_SITE -from invenio.legacy.bibdocfile.api import BibRecDocs, get_superformat_from_format -from invenio.config import CFG_WEBSEARCH_ENABLE_OPENGRAPH - -def format_element(bfo, max_photos='', one_icon_per_bibdoc='yes', twitter_card_type='photo', use_webjournal_featured_image='no'): - """Return an image of the record, suitable for the Open Graph protocol. - - Will look for any icon stored with the record, and will fallback to any - image file attached to the record. Returns nothing when no image is found. - - Some optional structured properties are not considered, for optimizing both generation of the page - and page size. - - @param max_photos: the maximum number of photos to display - @param one_icon_per_bibdoc: shall we keep just one icon per bibdoc in the output (not repetition of same preview in multiple sizes)? - @param twitter_card_type: the type of Twitter card: 'photo' (single photo) or 'gallery'. Fall back to 'photo' if not enough pictures for a 'gallery'. - @param use_webjournal_featured_image: if 'yes', use the "featured image" (as defined in bfe_webjournal_articles_overview) as image for the Twitter Card - """ - if not CFG_WEBSEARCH_ENABLE_OPENGRAPH: - return "" - bibarchive = BibRecDocs(bfo.recID) - bibdocs = bibarchive.list_bibdocs() - tags = [] - images = [] - - if max_photos.isdigit(): - max_photos = int(max_photos) - else: - max_photos = len(bibdocs) - - for doc in bibdocs[:max_photos]: - found_icons = [] - found_image_url = '' - found_image_size = 0 - for docfile in doc.list_latest_files(list_hidden=False): - if docfile.is_icon(): - found_icons.append((docfile.get_size(), docfile.get_url())) - elif get_superformat_from_format(docfile.get_format()).lower() in [".jpg", ".gif", ".jpeg", ".png"]: - found_image_url = docfile.get_url() - found_image_size = docfile.get_size() - found_icons.sort() - - # We might have found several icons for the same file: keep - # middle-size one - if found_icons: - if one_icon_per_bibdoc.lower() == 'yes': - found_icons = [found_icons[len(found_icons)/2]] - for icon_size, icon_url in found_icons: - images.append((icon_url, icon_url.replace(CFG_SITE_URL, CFG_SITE_SECURE_URL), icon_size)) - # Link to main file too (?) - if found_image_url: - images.append((found_image_url, found_image_url.replace(CFG_SITE_URL, CFG_SITE_SECURE_URL), found_image_size)) - - if CFG_CERN_SITE: - # Add some more pictures from metadata - dummy_size = 500*1024 # we don't we to check image size, we just make one (see Twitter Card limit) - additional_images = [(image_url, image_url.replace("http://mediaarchive.cern.ch/", "https://mediastream.cern.ch"), dummy_size) for image_url in bfo.fields("8567_u") if image_url.split('.')[-1] in ('jpg', 'png', 'jpeg', 'gif') and 'A5' in image_url] - images.extend(additional_images) - - tags = ['%s' % (image_url, image_url != image_secure_url and '\n' % image_secure_url or "") for image_url, image_secure_url, image_size in images] - - # Twitter Card - - if use_webjournal_featured_image.lower() == 'yes': - # First look for the prefered image, if available. Note that - # it might be a remote one. - try: - from invenio.modules.formatter.format_elements import bfe_webjournal_articles_overview - image_url = bfe_webjournal_articles_overview._get_feature_image(bfo) - image_secure_url = image_url.replace('http:', 'https:') - image_size = 500 * 1024 # TODO: check for real image size - if image_url.strip(): - images.insert(0, (image_url, image_secure_url, image_size)) - except: - pass - - # Filter out images that would not be compatible - twitter_compatible_images = [image_url for image_url, image_secure_url, image_size in images if \ - image_size < 1024*1024][:4] #Max 1MB according to Twitter Card APIs, max 4 photos - twitter_card_tags = [] - if len(twitter_compatible_images) == 4 and twitter_card_type == 'gallery': - twitter_card_tags = ['' % \ - (twitter_compatible_images.index(image_url), image_url) \ - for image_url in twitter_compatible_images] - elif twitter_compatible_images: - twitter_card_tags = ['' % twitter_compatible_images[0]] - - tags = twitter_card_tags + tags - - return "\n".join(tags) - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_meta_opengraph_video.py b/invenio/modules/formatter/format_elements/bfe_meta_opengraph_video.py deleted file mode 100644 index c2d14c60e..000000000 --- a/invenio/modules/formatter/format_elements/bfe_meta_opengraph_video.py +++ /dev/null @@ -1,109 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - return the video of a record""" - -import cgi -from invenio.config import CFG_SITE_URL, CFG_SITE_SECURE_URL, CFG_CERN_SITE -from invenio.legacy.bibdocfile.api import BibRecDocs, get_superformat_from_format -from invenio.config import CFG_WEBSEARCH_ENABLE_OPENGRAPH - -def format_element(bfo): - """ - Return the video of the record, suitable for the Open Graph protocol. - """ - if not CFG_WEBSEARCH_ENABLE_OPENGRAPH: - return "" - bibarchive = BibRecDocs(bfo.recID) - bibdocs = bibarchive.list_bibdocs() - additional_tags = "" - tags = [] - videos = [] - images = [] - - for doc in bibdocs: - found_icons = [] - found_image_url = '' - for docfile in doc.list_latest_files(): - if docfile.is_icon(): - found_icons.append((docfile.get_size(), docfile.get_url())) - elif get_superformat_from_format(docfile.get_format()).lower() in [".mp4", '.webm', '.ogv']: - found_image_url = docfile.get_url() - found_icons.sort() - - for icon_size, icon_url in found_icons: - images.append((icon_url, icon_url.replace(CFG_SITE_URL, CFG_SITE_SECURE_URL))) - if found_image_url: - videos.append((found_image_url, found_image_url.replace(CFG_SITE_URL, CFG_SITE_SECURE_URL))) - - if CFG_CERN_SITE: - mp4_urls = [url.replace('http://mediaarchive.cern.ch', 'https://mediastream.cern.ch') \ - for url in bfo.fields('8567_u') if url.endswith('.mp4')] - img_urls = [url.replace('http://mediaarchive.cern.ch', 'https://mediastream.cern.ch') \ - for url in bfo.fields('8567_u') if url.endswith('.jpg') or url.endswith('.png')] - - if mp4_urls: - mp4_url = mp4_urls[0] - if "4/3" in bfo.field("300__b"): - width = "640" - height = "480" - else: - width = "640" - height = "360" - additional_tags += ''' - - - - - - - - - - - - - ''' % {'CFG_CERN_PLAYER_URL': "https://cds.cern.ch/mediaplayer.swf", - 'CFG_STREAMER_URL': "rtmp://wowza.cern.ch:1935/vod", - 'width': width, - 'height': height, - 'image_url': img_urls and img_urls[0] or '', - 'mp4_url': mp4_url.replace('http://mediaarchive.cern.ch', 'https://mediastream.cern.ch'), - 'mp4_url_relative': '/' + '/'.join(mp4_url.split('/')[4:])} - try: - from invenio.media_utils import generate_embedding_url - embed_url = generate_embedding_url(bfo.field('037__a')) - additional_tags += '''''' % cgi.escape(embed_url, quote=True).replace('http://', 'https://', 1) - except: - pass - - - tags = ['%s' % (image_url, image_url != image_secure_url and '\n' % image_secure_url or "") for image_url, image_secure_url in images] - tags.extend(['%s' % (video_url, video_url != video_secure_url and '\n' % video_secure_url or "") for video_url, video_secure_url in videos]) - - - return "\n".join(tags) + additional_tags - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_notes.py b/invenio/modules/formatter/format_elements/bfe_notes.py deleted file mode 100644 index fe12183e7..000000000 --- a/invenio/modules/formatter/format_elements/bfe_notes.py +++ /dev/null @@ -1,71 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints notes -""" -__revision__ = "$Id$" - -import cgi - -def format_element(bfo, note_suffix, note_prefix='Note: ', separator='; '): - """ - Displays notes (various note fields) - - @param note_prefix: a prefix before each group of notes - @param note_suffix: a suffix after each group of notes - @param separator: a separator between notes of a group - """ - notes = [] - - notes_group_1 = bfo.fields('594__p') - if len(notes_group_1) > 0: - notes_group_1 = separator.join(notes_group_1) - notes.append(notes_group_1) - - notes_group_2 = bfo.fields('500__a') - if len(notes_group_2) > 0: - notes_group_2 = separator.join(notes_group_2) - notes.append(notes_group_2) - - - notes_group_3 = bfo.fields('502__a') - notes_group_3.extend(bfo.fields('909CCr')) - notes_group_3.extend(bfo.fields('909CPn')) - notes_group_3.extend(bfo.fields('711__a')) - if len(notes_group_3) > 0: - notes_group_3 = separator.join(notes_group_3) - notes.append(notes_group_3) - - notes_group_4 = bfo.fields('596__a') - - if len(notes_group_4) > 0: - notes_group_4 = separator.join(notes_group_4) - notes.append(notes_group_4) - - if len(notes) > 0: - - notes = [note_prefix + cgi.escape(x) + note_suffix - for x in notes] - return "".join(notes) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_oai_identifier.py b/invenio/modules/formatter/format_elements/bfe_oai_identifier.py deleted file mode 100644 index 33ae96e6d..000000000 --- a/invenio/modules/formatter/format_elements/bfe_oai_identifier.py +++ /dev/null @@ -1,40 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2012 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints record OAI identifier -""" -import cgi -from invenio.config import CFG_OAI_ID_FIELD - -def format_element(bfo, instance_prefix="", separator=", ", instance_suffix=""): - """ - Prints the record OAI identifier(s). - - @param instance_prefix: some value printed before each identifier. Must be already escaped - @param separator: some value printed between each identifier. Must be already escaped - @param instance_suffix: some value printed after each identifier. Must be already escaped - """ - return separator.join([instance_prefix + cgi.escape(value) + instance_suffix \ - for value in bfo.fields(CFG_OAI_ID_FIELD) if value]) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_oai_marcxml.py b/invenio/modules/formatter/format_elements/bfe_oai_marcxml.py deleted file mode 100644 index 47b874ee6..000000000 --- a/invenio/modules/formatter/format_elements/bfe_oai_marcxml.py +++ /dev/null @@ -1,49 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - OAI ready MARCXML - -This element return the full MARCXML representation of a record with the marc -prefix and namespace and adding the leader. -""" - -from invenio.modules.formatter.api import get_preformatted_record - -def format_element(bfo): - """ - Return the MARCXML representation of the record with the marc prefix and - namespace and adding the leader. - """ - formatted_record = get_preformatted_record(bfo.recID, 'xm') - formatted_record = formatted_record.replace("", "\n 00000coc 2200000uu 4500") - formatted_record = formatted_record.replace("", "\n 00000coc 2200000uu 4500") - formatted_record = formatted_record.replace("' - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_photos.py b/invenio/modules/formatter/format_elements/bfe_photos.py deleted file mode 100644 index c7d32a0c6..000000000 --- a/invenio/modules/formatter/format_elements/bfe_photos.py +++ /dev/null @@ -1,129 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Print photos of the record (if bibdoc file) -""" - -import cgi -from invenio.legacy.bibdocfile.api import BibRecDocs -from invenio.utils.url import (create_html_link, - get_relative_url) - - -def format_element(bfo, separator=" ", style='', img_style='', text_style='font-size:small', - print_links='yes', max_photos='', show_comment='yes', - img_max_width='250px', display_all_version_links='yes'): - """ - Lists the photos of a record. Display the icon version, linked to - its original version. - - This element works for photos appended to a record as BibDoc - files, for which a preview icon has been generated. If there are - several formats for one photo, use the first one found. - - @param separator: separator between each photo - @param print_links: if 'yes', print links to the original photo - @param style: style attributes of the whole image block. Eg: "padding:2px;border:1px" - @param img_style: style attributes of the images. Eg: "width:50px;border:none" - @param text_style: style attributes of the text. Eg: "font-size:small" - @param max_photos: the maximum number of photos to display - @param show_comment: if 'yes', display the comment of each photo - @param display_all_version_links: if 'yes', print links to additional (sub)formats - """ - photos = [] - bibarchive = BibRecDocs(bfo.recID) - bibdocs = bibarchive.list_bibdocs() - - if max_photos.isdigit(): - max_photos = int(max_photos) - else: - max_photos = len(bibdocs) - - for doc in bibdocs[:max_photos]: - found_icons = [] - found_url = '' - for docfile in doc.list_latest_files(): - if docfile.is_icon(): - found_icons.append(( - docfile.get_size(), - get_relative_url(docfile.get_url()) - )) - else: - found_url = get_relative_url(docfile.get_url()) - found_icons.sort() - - if found_icons: - additional_links = '' - name = bibarchive.get_docname(doc.id) - comment = doc.list_latest_files()[0].get_comment() - - preview_url = None - if len(found_icons) > 1: - preview_url = get_relative_url(found_icons[1][1]) - additional_urls = [(docfile.get_size(), get_relative_url(docfile.get_url()), \ - docfile.get_superformat(), docfile.get_subformat()) \ - for docfile in doc.list_latest_files() if not docfile.is_icon()] - additional_urls.sort() - additional_links = [create_html_link(url, urlargd={}, \ - linkattrd={'style': 'font-size:x-small'}, \ - link_label="%s %s (%s)" % (format.strip('.').upper(), subformat, format_size(size))) \ - for (size, url, format, subformat) in additional_urls] - img = '%(name)s' % \ - {'icon_url': cgi.escape(get_relative_url(found_icons[0][1]), True), - 'name': cgi.escape(name, True), - 'img_style': img_style, - 'img_max_width': img_max_width} - - if print_links.lower() == 'yes': - img = '%s' % (cgi.escape(preview_url or found_url, True), img) - if display_all_version_links.lower() == 'yes' and additional_links: - img += '
    ' + ' '.join(additional_links) + '
    ' - - if show_comment.lower() == 'yes' and comment: - img += '
    %(comment)s
    ' % \ - {'comment': comment.replace('\n', '
    '), - 'text_style': text_style} - img = '
    %(img)s
    ' % \ - {'img_max_width': img_max_width, - 'style': style, - 'img': img} - - photos.append(img) - - return '
    ' + separator.join(photos) + '
    ' - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - -def format_size(size): - """ - Get human-readable string for the given size in Bytes - """ - if size < 1024: - return "%d byte%s" % (size, size != 1 and 's' or '') - elif size < 1024 * 1024: - return "%.1f KB" % (size / 1024) - elif size < 1024 * 1024 * 1024: - return "%.1f MB" % (size / (1024 * 1024)) - else: - return "%.1f GB" % (size / (1024 * 1024 * 1024)) diff --git a/invenio/modules/formatter/format_elements/bfe_place.py b/invenio/modules/formatter/format_elements/bfe_place.py deleted file mode 100644 index 547ab95fc..000000000 --- a/invenio/modules/formatter/format_elements/bfe_place.py +++ /dev/null @@ -1,35 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints imprint publication place -""" -__revision__ = "$Id$" - -def format_element(bfo): - """ - Prints the imprint publication place as HTML - @see: date.py, publisher.py, reprints.py, imprint.py, pagination.py - """ - place = bfo.field('260__a') - - if place != "sine loco": - return place - - - - diff --git a/invenio/modules/formatter/format_elements/bfe_plots.py b/invenio/modules/formatter/format_elements/bfe_plots.py deleted file mode 100644 index 83ae7de15..000000000 --- a/invenio/modules/formatter/format_elements/bfe_plots.py +++ /dev/null @@ -1,107 +0,0 @@ -# -*- coding: utf-8 -*- -# -# $Id: bfe_CERN_plots.py,v 1.3 2009/03/17 10:55:15 jerome Exp $ -# -# This file is part of Invenio. -# Copyright (C) 2010, 2011, 2013, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Display image of the plot if we are in selected plots collection -""" - -from invenio.legacy.bibdocfile.api import BibRecDocs -from invenio.utils.url import create_html_link -from invenio.config import CFG_SITE_RECORD -from invenio.base.i18n import gettext_set_language - -try: - from invenio.config import CFG_BASE_URL -except ImportError: - from invenio.config import CFG_SITE_URL - CFG_BASE_URL = CFG_SITE_URL - - -def format_element(bfo, width="", caption="yes", max_plots="3"): - """ - Display image of the plot if we are in selected plots collections - - To achieve this, we take the pngs associated with this document - - @param width: the width of the returned image (Eg: '100px') - @param caption: display the captions or not? - @param max_plots: the maximum number of plots to display (-1 is all plots) - """ - _ = gettext_set_language(bfo.lang) - - img_files = [] - try: - max_plots = int(max_plots) - except ValueError: - # Someone tried to squeeze in something non-numerical. Hah! - max_plots = 3 - - link = "" - bibarchive = BibRecDocs(bfo.recID) - - if width != "": - width = 'width="%s"' % width - - for doc in bibarchive.list_bibdocs(doctype="Plot"): - for _file in doc.list_latest_files(): - if _file.subformat == "context": - # Ignore context files - continue - - caption_text = _file.get_description()[5:] - index = int(_file.get_description()[:5]) - img_location = _file.get_url() - - img = '' % \ - (img_location, caption_text, width) - - plotlink = create_html_link(urlbase='%s/%s/%s/plots#%d' % - (CFG_BASE_URL, - CFG_SITE_RECORD, - bfo.recID, - index), - urlargd={}, - link_label=img) - - img_files.append((index, plotlink)) - - img_files = sorted(img_files, key=lambda x: x[0]) - if max_plots > 0: - img_files = img_files[:max_plots] - - if len(img_files) >= max_plots: - link = "%s" % \ - (CFG_SITE_RECORD, bfo.recID, _("Show more plots")) - - for index in range(len(img_files)): - img_files[index] = img_files[index][1] - - if len(img_files) == 0: - return '' - - return '
    ' +\ - " ".join(img_files) + ' ' + link + '
    ' - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_plots_thumb.py b/invenio/modules/formatter/format_elements/bfe_plots_thumb.py deleted file mode 100644 index 5a14f0a68..000000000 --- a/invenio/modules/formatter/format_elements/bfe_plots_thumb.py +++ /dev/null @@ -1,70 +0,0 @@ -# -*- coding: utf-8 -*- -# -# $Id: bfe_CERN_plots.py,v 1.3 2009/03/17 10:55:15 jerome Exp $ -# -# This file is part of Invenio. -# Copyright (C) 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Display image of the plot if we are in selected plots collection -""" -__revision__ = "$Id: bfe_CERN_plots.py,v 1.3 2009/03/17 10:55:15 jerome Exp $" - -from invenio.legacy.bibdocfile.api import BibRecDocs - - -def format_element(bfo): - """ - Display image of the thumbnail plot if we are in selected plots collections - """ - ## To achieve this, we take the Thumb file associated with this document - - bibarchive = BibRecDocs(bfo.recID) - - img_files = [] - - for doc in bibarchive.list_bibdocs(): - for _file in doc.list_latest_files(): - if _file.get_type() == "Plot": - caption_text = _file.get_description()[5:] - index = int(_file.get_description()[:5]) - img_location = _file.get_url() - - if img_location == "": - continue - - img = '' % \ - (img_location) - img_files.append((index, img_location)) # FIXME: was link here - - if _file.get_type() == "Thumb": - img_location = _file.get_url() - img = '' % \ - (img_location) - return '
    ' + img + '
    ' - - # then we use the default: the last plot with an image - img_files = sorted(img_files, key=lambda x: x[0]) - if img_files: - return '
    ' + img_files[-1][1] + '
    ' - else: - return '' - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_publi_info.py b/invenio/modules/formatter/format_elements/bfe_publi_info.py deleted file mode 100644 index 72670d260..000000000 --- a/invenio/modules/formatter/format_elements/bfe_publi_info.py +++ /dev/null @@ -1,97 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints publcation information and link to ejournal -""" -__revision__ = "$Id$" - -from urllib import quote -import cgi - -def format_element(bfo): - """ - Displays inline publication information with html link to ejournal - (when available). - """ - - - out = '' - - publication_info = bfo.field('909C4') - if publication_info == "": - return "" - - journal_source = publication_info.get('p') - journal = bfo.kb('ejournals', journal_source) - volume = publication_info.get('v') - year = publication_info.get('y') - number = publication_info.get('n') - pages = publication_info.get('c') - doi = publication_info.get('a') - - if journal is not None: - journal = cgi.escape(journal) - if volume is not None: - volume = cgi.escape(volume) - if year is not None: - year = cgi.escape(year) - if number is not None: - number = cgi.escape(number) - if pages is not None: - pages = cgi.escape(pages) - if doi is not None: - doi = cgi.escape(doi) - - if journal != '' and volume is not None: - - out += '%(journal)s :%(volume)s %(year)s %(page)s' % {'journal': journal, - 'volume': volume, - 'year': year, - 'page': pages} - else: - out += journal_source + ': ' - if volume is not None: - out += volume - if year is not None: - out += ' (' + year + ') ' - if number is not None: - out += 'no. ' + number + ', ' - if pages is not None: - out += 'pp. ' + pages - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - - - - diff --git a/invenio/modules/formatter/format_elements/bfe_publisher.py b/invenio/modules/formatter/format_elements/bfe_publisher.py deleted file mode 100644 index fdf87e4ee..000000000 --- a/invenio/modules/formatter/format_elements/bfe_publisher.py +++ /dev/null @@ -1,54 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints publisher name -""" -__revision__ = "$Id$" - -from invenio.config import CFG_SITE_URL - -from invenio.legacy.bibauthority.engine import get_low_level_recIDs_from_control_no - -def format_element(bfo): - """ - Prints the publisher name - - @see: place.py, date.py, reprints.py, imprint.py, pagination.py - """ - - publisher = bfo.field('260__b') - control_no = bfo.field('260__0') - - if publisher != "sine nomine": - if control_no: - recIDs = get_low_level_recIDs_from_control_no(control_no) - if len(recIDs): - publisher = '' + publisher + '' - return publisher - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - diff --git a/invenio/modules/formatter/format_elements/bfe_qrcode.py b/invenio/modules/formatter/format_elements/bfe_qrcode.py deleted file mode 100644 index fa18a8836..000000000 --- a/invenio/modules/formatter/format_elements/bfe_qrcode.py +++ /dev/null @@ -1,83 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - QR code generator """ - -from invenio.config import CFG_SITE_SECURE_URL, CFG_WEBDIR, CFG_SITE_RECORD -from invenio.utils.hash import md5 -import os -try: - import qrcode - from PIL import Image - HAS_QR = True -except ImportError: - HAS_QR = False - -if not HAS_QR: - from warnings import warn - warn("Please install qrcode and " - "Pillow (Python Imaging Library) " - ".") - - -def _get_record_hash(link): - """ - Generate a record hash including CFG_SITE_URL so that - if CFG_SITE_URL is updated, the QR-code image is invalidated. - """ - return md5(link).hexdigest()[:8].lower() - - -def format_element(bfo, width="100"): - """ - Generate a QR-code image linking to the current record. - - @param width: Width of QR-code image. - """ - if not HAS_QR: - return "" - - width = int(width) - - bibrec_id = bfo.control_field("001") - link = "%s/%s/%s" % (CFG_SITE_SECURE_URL, CFG_SITE_RECORD, bibrec_id) - hash_val = _get_record_hash(link) - - filename = "%s_%s.png" % (bibrec_id, hash_val) - filename_url = "/img/qrcodes/%s" % filename - filename_path = os.path.join(CFG_WEBDIR, "img/qrcodes/%s" % filename) - - if not os.path.exists(filename_path): - if not os.path.exists(os.path.dirname(filename_path)): - os.makedirs(os.path.dirname(filename_path)) - - img = qrcode.make(link) - img._img = img._img.convert("RGBA") - img._img = img._img.resize((width, width), Image.ANTIALIAS) - img.save(filename_path, "PNG") - - return """""" % (filename_url, width) - - -def escape_values(dummy_bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_record_id.py b/invenio/modules/formatter/format_elements/bfe_record_id.py deleted file mode 100644 index 419a7f359..000000000 --- a/invenio/modules/formatter/format_elements/bfe_record_id.py +++ /dev/null @@ -1,25 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - Print the record ID.""" - - -def format_element(bfo): - """Print the record id.""" - return bfo.control_field('001') diff --git a/invenio/modules/formatter/format_elements/bfe_record_stats.py b/invenio/modules/formatter/format_elements/bfe_record_stats.py deleted file mode 100644 index 05ecbcb00..000000000 --- a/invenio/modules/formatter/format_elements/bfe_record_stats.py +++ /dev/null @@ -1,56 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints record statistics -""" -__revision__ = "$Id$" - -from invenio.legacy.dbquery import run_sql - -def format_element(bfo, display='day_distinct_ip_nb_views'): - ''' - Prints record statistics - - @param display: the type of statistics displayed. Can be 'total_nb_view', 'day_nb_views', 'total_distinct_ip_nb_views', 'day_distincts_ip_nb_views', 'total_distinct_ip_per_day_nb_views' - ''' - - if display == 'total_nb_views': - return run_sql("""SELECT COUNT(client_host) FROM rnkPAGEVIEWS - WHERE id_bibrec=%s""", - (bfo.recID,))[0][0] - elif display == 'day_nb_views': - return run_sql("""SELECT COUNT(client_host) FROM rnkPAGEVIEWS - WHERE id_bibrec=%s AND DATE(view_time)=CURDATE()""", - (bfo.recID,))[0][0] - elif display == 'total_distinct_ip_nb_views': - return run_sql("""SELECT COUNT(DISTINCT client_host) FROM rnkPAGEVIEWS - WHERE id_bibrec=%s""", - (bfo.recID,))[0][0] - elif display == 'day_distinct_ip_nb_views': - return run_sql("""SELECT COUNT(DISTINCT client_host) FROM rnkPAGEVIEWS - WHERE id_bibrec=%s AND DATE(view_time)=CURDATE()""", - (bfo.recID,))[0][0] - elif display == 'total_distinct_ip_per_day_nb_views': - # Count the number of distinct IP addresses for every day Then - # sum up. Similar to total_distinct_users_nb_views but assume - # that several different users can be behind a single IP - # (which could change every day) - res = run_sql("""SELECT COUNT(DISTINCT client_host) - FROM rnkPAGEVIEWS - WHERE id_bibrec=%s GROUP BY DATE(view_time)""", - (bfo.recID,)) - return sum([row[0] for row in res]) - diff --git a/invenio/modules/formatter/format_elements/bfe_record_url.py b/invenio/modules/formatter/format_elements/bfe_record_url.py deleted file mode 100644 index ddd1ca12f..000000000 --- a/invenio/modules/formatter/format_elements/bfe_record_url.py +++ /dev/null @@ -1,40 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the record URL. -""" - -__revision__ = "$Id$" - -from invenio.config import \ - CFG_SITE_URL, \ - CFG_SITE_RECORD - - -def format_element(bfo, with_ln="yes"): - """ - Prints the record URL. - - @param with_ln: if "yes" include "ln" attribute in the URL - """ - url = CFG_SITE_URL + "/" + CFG_SITE_RECORD + "/" + bfo.control_field('001') - - if with_ln.lower() == "yes": - url += "?ln=" + bfo.lang - - return url diff --git a/invenio/modules/formatter/format_elements/bfe_references.py b/invenio/modules/formatter/format_elements/bfe_references.py deleted file mode 100644 index 92eb14650..000000000 --- a/invenio/modules/formatter/format_elements/bfe_references.py +++ /dev/null @@ -1,85 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - Print references.""" - - -def format_element(bfo, reference_prefix, reference_suffix): - """Print the references of this record. - - :param reference_prefix: a prefix displayed before each reference - :param reference_suffix: a suffix displayed after each reference - """ - from invenio.config import CFG_BASE_URL - - references = bfo.fields("999C5", escape=1) - out = "" - - for reference in references: - ref_out = '' - - if 'o' in reference: - if out != "": - ref_out = '' - ref_out += "
  • "+ reference['o']+ " " - - if 'm' in reference: - ref_out += ""+ reference['m']+ " " - - if 'r' in reference: - ref_out += ' ['+ reference['r']+ "]
    " - - if 't' in reference: - ejournal = bfo.kb("ejournals", reference.get('t', "")) - if ejournal != "": - ref_out += ' ' - ref_out += reference['t']+": "+reference.get('v', "")+\ - " ("+reference.get('y', "")+") " - ref_out += reference.get('p', "")+"
    " - else: - ref_out += " "+reference['t']+ reference.get('v', "")+\ - reference.get('y',"")+ reference.get('p',"")+ \ - "
    " - - - if reference_prefix is not None and ref_out != '': - ref_out = reference_prefix + ref_out - if reference_suffix is not None and ref_out != '': - ref_out += reference_suffix - - out += ref_out - - if out != '': - out += '
    • ' - - return out - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_report_numbers.py b/invenio/modules/formatter/format_elements/bfe_report_numbers.py deleted file mode 100644 index 56df33331..000000000 --- a/invenio/modules/formatter/format_elements/bfe_report_numbers.py +++ /dev/null @@ -1,70 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints report numbers -""" - -__revision__ = "" - -import cgi -from invenio.utils.url import create_html_link - -def format_element(bfo, limit, separator=" ", extension=" etc.", link='yes', just_one='no'): - """ - Prints the report numbers of the record (037__a and 088__a) - - @param separator: the separator between report numbers. - @param limit: the max number of report numbers to print - @param extension: a prefix printed when limit param is reached - @param link: if 'yes', display report number with corresponding link when possible - """ - numbers = bfo.fields("037__a") - numbers.extend(bfo.fields("088__a")) - - # Only display the first one - if just_one == 'yes': - numbers = numbers[:1] - - if limit.isdigit(): - limit_as_int = int(limit) - if limit_as_int <= len(numbers): - return separator.join(numbers[:limit_as_int]) + extension - - return separator.join([build_report_number_link(report_number, \ - link == 'yes') \ - for report_number in numbers]) - -def build_report_number_link(report_number, link_p=True): - """ - Build HTML link out of given report number when it make sense (or - is possible) and/or escape report number. - @param report_number: the report number to consider - @param link_p: if True, build link, otherwise just escape - """ - if link_p and report_number.lower().startswith('arxiv:'): - return create_html_link('http://arxiv.org/abs/' + report_number, - urlargd={}, link_label=report_number) - else: - return cgi.escape(report_number) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_reprints.py b/invenio/modules/formatter/format_elements/bfe_reprints.py deleted file mode 100644 index 9a657bb96..000000000 --- a/invenio/modules/formatter/format_elements/bfe_reprints.py +++ /dev/null @@ -1,31 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints reprinted editions -""" -__revision__ = "$Id$" - -def format_element(bfo, separator): - """ - Prints the reprinted editions of a record - - @param separator: a separator between reprinted editions - @see: place.py, publisher.py, imprint.py, date.py, pagination.py - """ - - reprints = bfo.field('260__g') - if len(reprints) > 0: - return separator.join(reprints) diff --git a/invenio/modules/formatter/format_elements/bfe_sciencewise.py b/invenio/modules/formatter/format_elements/bfe_sciencewise.py deleted file mode 100644 index 2eb063c4e..000000000 --- a/invenio/modules/formatter/format_elements/bfe_sciencewise.py +++ /dev/null @@ -1,115 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - ScienceWise.info - -This elements displays a linking icon to ScienceWise.info for arXiv -records. -""" - -import cgi -import re - -from invenio.config import CFG_BASE_URL, CFG_SITE_LANG, CFG_CERN_SITE -from invenio.base.i18n import gettext_set_language - -_RE_MODERN_ARXIV = re.compile('(arxiv:)?(?P\d{4}.\d{4}(v\d+)?)') -_RE_OLD_ARXIV = re.compile('(arxiv:)?(?P\w+-\w+/\d{7}(v\d+)?)') -_RE_BAD_OLD_ARXIV = re.compile('(arxiv:)?(?P\w+-\w+)-(?P\d{7}(v\d+)?)') - -def format_element(bfo): - """ - If the record has an ArXiv reportnumber, displays a ScienceWise icon - to bookmark it. - """ - _ = gettext_set_language(bfo.lang) - for tag in ('037__a', '088__a'): - for reportnumber in bfo.fields(tag): - icon = create_sciencewise_icon(reportnumber) - if icon: - return icon - if CFG_CERN_SITE: - return create_sciencewise_icon(bfo.recID, cds=True) - return "" - -def get_arxiv_reportnumber(bfo): - """ - Return an ArXiv reportnumber (if any) from the corresponding record. - Return empty string otherwise. - """ - for tag in ('037__a', '088__a'): - for reportnumber in bfo.fields(tag): - reportnumber = reportnumber.lower() - for regexp in (_RE_MODERN_ARXIV, _RE_OLD_ARXIV): - g = regexp.match(reportnumber) - if g: - return g.group('number') - return "" - - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - -def create_sciencewise_url(reportnumber, cds=False): - """ - If the reportnumber is a valid arXiv reportnumber return a ScienceWise.info - URL. - """ - if cds: - return "http://sciencewise.info/bookmarks/cds:%s/add" % reportnumber - reportnumber = reportnumber.lower() - g = _RE_BAD_OLD_ARXIV.match(reportnumber) - if g: - reportnumber = '%s/%s' % (g.group('archive'), g.group('number')) - for regexp in (_RE_MODERN_ARXIV, _RE_OLD_ARXIV): - g = regexp.match(reportnumber) - if g: - return "http://sciencewise.info/bookmarks/%s/add" % g.group('number') - return "" - -def create_sciencewise_icon(reportnumber, lang=CFG_SITE_LANG, cds=False): - """ - If the reportnumber is a valid arXiv reportnumber return a ScienceWise.info - icon. - """ - _ = gettext_set_language(lang) - if cds: - return """\ - ScienceWise.info icon""" % { - 'id': cgi.escape(reportnumber, True), - 'title': cgi.escape(_("Add this document to your ScienceWise.info bookmarks"), True), - 'siteurl': cgi.escape(CFG_BASE_URL, True) - } - reportnumber = reportnumber.lower() - g = _RE_BAD_OLD_ARXIV.match(reportnumber) - if g: - reportnumber = '%s/%s' % (g.group('archive'), g.group('number')) - for regexp in (_RE_MODERN_ARXIV, _RE_OLD_ARXIV): - g = regexp.match(reportnumber) - if g: - return """\ - ScienceWise.info icon""" % { - 'id': cgi.escape(g.group('number'), True), - 'title': cgi.escape(_("Add this article to your ScienceWise.info bookmarks"), True), - 'siteurl': cgi.escape(CFG_BASE_URL, True) - } - return "" diff --git a/invenio/modules/formatter/format_elements/bfe_server_info.py b/invenio/modules/formatter/format_elements/bfe_server_info.py deleted file mode 100644 index ba54db2a6..000000000 --- a/invenio/modules/formatter/format_elements/bfe_server_info.py +++ /dev/null @@ -1,100 +0,0 @@ -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints server info -""" - -import urllib -import re -from invenio.config import CFG_SITE_URL, CFG_BASE_URL, CFG_SITE_ADMIN_EMAIL, \ - CFG_SITE_LANG, CFG_SITE_NAME, CFG_VERSION, CFG_SITE_NAME_INTL, \ - CFG_SITE_SUPPORT_EMAIL, CFG_SITE_RECORD, CFG_BIBUPLOAD_INTERNAL_DOI_PATTERN -from invenio.legacy.bibrecord import record_extract_dois - - -def format_element(bfo, var=''): - ''' - Print several server specific variables. - @param var: the name of the desired variable. Can be one of: CFG_SITE_NAME, CFG_SITE_NAME_INTL, CFG_SITE_LANG, CFG_VERSION, CFG_SITE_ADMIN_EMAIL, CFG_SITE_SUPPORT_EMAIL, CFG_SITE_URL, searchurl, recurl - CFG_SITE_NAME: the name of the server - CFG_SITE_NAME_INTL: internationalized name - CFG_SITE_LANG: the default language of the server - CFG_VERSION: the software version - CFG_SITE_ADMIN_EMAIL: the admin email - CFG_SITE_SUPPORT_EMAIL: the support email - CFG_SITE_URL: the base url for the server - searchurl: the search url for the server - recurl: the base url for the record - recinternaldoiurl_or_recurl: the base url for the record, as an internal CFG_SITE_URL + '/doi' URL when DOI is available and matched by CFG_BIBUPLOAD_INTERNAL_DOI_PATTERN - ''' - dois = [] - if var == 'recinternaldoiurl_or_recurl': - # Prepare list of internal DOIs of that record - dois = [doi for doi in record_extract_dois(bfo.get_record()) if \ - re.compile(CFG_BIBUPLOAD_INTERNAL_DOI_PATTERN).match(doi)] - - recID = bfo.recID - if var == '': - out = '' - elif var in ['name', 'CFG_SITE_NAME']: - out = CFG_SITE_NAME - elif var in ['i18n_name', 'CFG_SITE_NAME_INTL']: - out = CFG_SITE_NAME_INTL.get(bfo.lang, CFG_SITE_NAME) - elif var in ['lang', 'CFG_SITE_LANG']: - out = CFG_SITE_LANG - elif var == 'CFG_VERSION': - out = 'Invenio v' + str(CFG_VERSION) - elif var in ['email', 'admin_email', 'CFG_SITE_ADMIN_EMAIL']: - out = CFG_SITE_ADMIN_EMAIL - elif var in ['support_email', 'CFG_SITE_SUPPORT_EMAIL']: - out = CFG_SITE_SUPPORT_EMAIL - elif var in ['CFG_SITE_RECORD']: - out = CFG_SITE_RECORD - elif var in ['weburl', 'CFG_SITE_URL']: - out = CFG_SITE_URL - if not out.endswith('/'): - out += '/' - elif var in ['CFG_BASE_URL']: - out = CFG_BASE_URL - if not out.endswith('/'): - out += '/' - elif var == 'searchurl': - out = CFG_BASE_URL + '/search' - if not out.endswith('/'): - out += '/' - elif var == 'absolutesearchurl': - out = CFG_SITE_URL + '/search' - if not out.endswith('/'): - out += '/' - elif var == 'absoluterecurl': - out = CFG_SITE_URL - if not out.endswith('/'): - out += '/' - out += CFG_SITE_RECORD + '/' + str(recID) - elif var == 'recinternaldoiurl_or_recurl' and \ - dois: - out = CFG_SITE_URL - if not out.endswith('/'): - out += '/' - out += 'doi/' + urllib.quote(dois[0]) - elif var in ('recurl', 'recinternaldoiurl_or_recurl'): - out = CFG_SITE_URL - if not out.endswith('/'): - out += '/' - out += CFG_SITE_RECORD +'/' + str(recID) - else: - out = 'Unknown variable: %s' % (var) - return out diff --git a/invenio/modules/formatter/format_elements/bfe_sword_push.py b/invenio/modules/formatter/format_elements/bfe_sword_push.py deleted file mode 100644 index 9b6b101f8..000000000 --- a/invenio/modules/formatter/format_elements/bfe_sword_push.py +++ /dev/null @@ -1,57 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Print link to push an entry to a remote server through SWORD -""" -__revision__ = "$Id$" - -from invenio.config import CFG_BASE_URL -from invenio.utils.url import create_html_link -from invenio.modules.access.engine import acc_authorize_action - -def format_element(bfo, remote_server_id, link_label="Push via Sword"): - """ - Print link to push an entry to a remote server through SWORD - - @param remote_server_id: ID of the remove server to link to. When - not specified, link to BibSword page - allowing to select server. - """ - user_info = bfo.user_info - auth_code, auth_message = acc_authorize_action(user_info, 'runbibswordclient') - if auth_code != 0: - return "" - - sword_arguments = {'ln': bfo.lang, - 'recid': bfo.recID} - - if remote_server_id: - sword_arguments['id_remote_server'] = remote_server_id - else: - sword_arguments['status'] = 'select_server' - - return create_html_link(CFG_BASE_URL + '/bibsword', - sword_arguments, - link_label) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_title.py b/invenio/modules/formatter/format_elements/bfe_title.py deleted file mode 100644 index d15904166..000000000 --- a/invenio/modules/formatter/format_elements/bfe_title.py +++ /dev/null @@ -1,89 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints titles -""" -__revision__ = "$Id$" - -import cgi - -def format_element(bfo, separator=" ", highlight='no'): - """ - Prints the titles of a record. - - @param separator: separator between the different titles - @param highlight: highlights the words corresponding to search query if set to 'yes' - """ - titles = [] - - title = bfo.field('245__a') - title_remainder = bfo.field('245__b') - edition_statement = bfo.field('250__a') - title_tome = bfo.field('245__n') - title_part = bfo.field('245__p') - - if len(title) > 0: - if title_remainder: - title += ': ' + title_remainder - if len(title_tome) > 0: - title += ", " + title_tome - if len(title_part) > 0: - title += ": " + title_part - titles.append( title ) - - title = bfo.field('246__a') - if len(title) > 0: - titles.append( title ) - - title = bfo.field('246__b') - if len(title) > 0: - titles.append( title ) - - title = bfo.field('246_1a') - if len(title) > 0: - titles.append( title ) - - if len(titles) > 0: - #Display 'Conference' title only if other titles were not found - title = bfo.field('111__a') - if len(title) > 0: - titles.append( title ) - - titles = [cgi.escape(x) for x in titles] - - if highlight == 'yes': - from invenio.modules.formatter import utils as bibformat_utils - titles = [bibformat_utils.highlight(x, bfo.search_pattern) for x in titles] - - if len(edition_statement) > 0: - return separator.join(titles) + "; " + edition_statement - else: - return separator.join(titles) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 - - - - - - diff --git a/invenio/modules/formatter/format_elements/bfe_title_brief.py b/invenio/modules/formatter/format_elements/bfe_title_brief.py deleted file mode 100644 index 89dd20e8d..000000000 --- a/invenio/modules/formatter/format_elements/bfe_title_brief.py +++ /dev/null @@ -1,62 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints short title -""" -__revision__ = "$Id$" - -def format_element(bfo, highlight="no", multilang='no'): - """ - Prints a short title, suitable for brief format. - - @param highlight: highlights the words corresponding to search query if set to 'yes' - """ - if multilang == 'yes': - if bfo.lang == 'fr': - title = bfo.field('246_1a') - else: - title = bfo.field('245__a') - else: - title = bfo.field('245__a') - title_remainder = bfo.field('245__b') - title_tome = bfo.field('245__n') - title_part = bfo.field('245__p') - edition_statement = bfo.field('250__a') - - out = title - if len(title_remainder) > 0: - out += ": " + title_remainder - if len(edition_statement) > 0: - out += "; " + edition_statement - if len(title_tome) > 0: - out += ", " + title_tome - if len(title_part) > 0: - out += ": " + title_part - - #Try to display 'Conference' title if other titles were not found - if out == '': - out += bfo.field('111__a') - - if highlight == 'yes': - from invenio.modules.formatter import utils as bibformat_utils - out = bibformat_utils.highlight(out, bfo.search_pattern, - prefix_tag="", - suffix_tag='') - - return out - diff --git a/invenio/modules/formatter/format_elements/bfe_topbanner.py b/invenio/modules/formatter/format_elements/bfe_topbanner.py deleted file mode 100644 index 7ac8d64b2..000000000 --- a/invenio/modules/formatter/format_elements/bfe_topbanner.py +++ /dev/null @@ -1,87 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -""" -BibFormat element - Prints HTML topbanner with category, rep. number, etc. -""" - -__revision__ = "$Id$" - -import cgi - -def format_element(bfo, kb_name="dbcollid2coll"): - """ - HTML top page banner containing category, rep. number, etc - """ - collection_indicator = bfo.kb(kb_name, bfo.field("980__a")) - collection_indicator = cgi.escape(collection_indicator) - subject = bfo.field("65017a") - subject = cgi.escape(subject) - subject_2 = bfo.field("65027a") - subject_2 = cgi.escape(subject_2) - additional_report_numbers = bfo.fields("088__a") - - source_of_aquisition = bfo.field("037__a") - source_of_aquisition = cgi.escape(source_of_aquisition) - - - if subject: - subject = " / " + subject - - if subject_2: - subject_2 = " / " + subject_2 - - if len(source_of_aquisition) > 0: - source_of_aquisition = '    " - - report_numbers_out = '' - for report_number in additional_report_numbers: - report_numbers_out += "" - - out = ''' -
    '+ source_of_aquisition + "" + \ - cgi.escape(report_number) + \ - "
    - - - - %(source_of_aquisition)s - -
    %(collection_indicator)s%(subject)s%(subject_2)s%(report_number)s
    - ''' % {'collection_indicator': collection_indicator, - 'subject': subject, - 'subject_2': subject_2, - 'report_number': report_numbers_out, - 'source_of_aquisition': source_of_aquisition} - - if collection_indicator or \ - subject or \ - subject_2 or \ - source_of_aquisition or \ - report_numbers_out: - return out - else: - return '' - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_url.py b/invenio/modules/formatter/format_elements/bfe_url.py deleted file mode 100644 index 567f0fa49..000000000 --- a/invenio/modules/formatter/format_elements/bfe_url.py +++ /dev/null @@ -1,44 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints full-text URLs -""" -__revision__ = "$Id$" - -def format_element(bfo, style, separator='; '): - """ - This is the default format for formatting full-text URLs. - @param separator: the separator between urls. - @param style: CSS class of the link - """ - - urls_u = bfo.fields("8564_u") - if style != "": - style = 'class="'+style+'"' - - urls = ['' + url +'' - for url in urls_u] - return separator.join(urls) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_video_bigthumb.py b/invenio/modules/formatter/format_elements/bfe_video_bigthumb.py deleted file mode 100644 index 18fa6e69c..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_bigthumb.py +++ /dev/null @@ -1,39 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Creates the element of the big thumbnail -""" - - -def format_element(bfo, z_field="POSTER"): - """ Called bz Bibformat to create the element. - """ - fields = bfo.fields('8564_') - if fields: - for field in fields: - if field.get('z') == z_field: - url = field.get('u', '') - alt = field.get('y', '') - return '\"%s\"/' % (url, alt) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return diff --git a/invenio/modules/formatter/format_elements/bfe_video_platform_downloads.py b/invenio/modules/formatter/format_elements/bfe_video_platform_downloads.py deleted file mode 100644 index 79741764f..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_platform_downloads.py +++ /dev/null @@ -1,99 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element -* Part of the Video Platform Prototype -* Generates a list of links to download videos directly -* The list includes the codec/container, subformat/resolution and file size -""" - -from invenio.legacy.bibdocfile.api import BibRecDocs - -html_skeleton_popup = """ -
    - %(elements)s -
    -""" - -html_skeleton_element = """ -
    - -
    - %(video_codec)s -
    -
    - %(video_size)s -
    -
    - %(video_resolution)s -
    -     -
    -""" - -def format_element(bfo): - """Format Element Function""" - return create_download_popup(bfo) - -def create_download_popup(bfo): - """Create the complete download popup""" - elements = [] - recdoc = BibRecDocs(bfo.recID) - bibdocs = recdoc.list_bibdocs() - ## Go through all the BibDocs and search for video related signatures - for bibdoc in bibdocs: - bibdocfiles = bibdoc.list_all_files() - for bibdocfile in bibdocfiles: - ## When a video signature is found, add it as an element - if bibdocfile.get_superformat() in ('.mp4', '.webm', '.ogv', - '.mov', '.wmv', '.avi', - '.mpeg', '.flv', '.mkv'): - url = bibdocfile.get_url() - codec = bibdocfile.get_superformat()[1:] - resolution = bibdocfile.get_subformat() - size = bibdocfile.get_size() - elements.append(create_download_element(url, codec, - size, resolution)) - if elements: - return html_skeleton_popup % { - 'elements': "\n".join(elements) - } - else: - return "" - -def create_download_element(url, codec, size, resolution): - """Creates an HTML element based on the element skeleton""" - return html_skeleton_element % { - 'video_url': url + "&download=1", - 'video_codec': codec.upper(), - 'video_size': human_size(size), - 'video_resolution': resolution - } - -def human_size(byte_size): - for x in ['bytes','KB','MB','GB','TB']: - if byte_size < 1024.0: - return "%3.1f %s" % (byte_size, x) - byte_size /= 1024.0 - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_video_platform_sources.py b/invenio/modules/formatter/format_elements/bfe_video_platform_sources.py deleted file mode 100644 index 5f99482b5..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_platform_sources.py +++ /dev/null @@ -1,112 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element -* Part of the video platform prototype -* Creates a - %s - - """ % '\n'.join(options) - return text - - -def create_option_element(width, height, subformat, url_webm=None, url_ogv=None, url_mp4=None, url_poster=None,): - """ Creates an HTML option element that carries all video information - """ - if url_webm: - webm = """data-src-webm="%s" data-type-webm='video/webm; codecs="vp8, vorbis"'""" % url_webm - else: - webm = "" - if url_ogv: - ogv = """data-src-ogg="%s" data-type-ogv='video/ogv; codecs="theora, vorbis"'""" % url_ogv - else: - ogv = "" - if url_mp4: - mp4 = """data-src-mp4="%s" data-type-mp4='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'""" % url_mp4 - else: - mp4 = "" - text = """""" % { - 'webm': webm, - 'ogv': ogv, - 'mp4': mp4, - 'url_poster': url_poster, - 'width': width, - 'height': height, - 'subformat': subformat - } - return text - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_video_platform_suggestions.py b/invenio/modules/formatter/format_elements/bfe_video_platform_suggestions.py deleted file mode 100644 index 94c133d75..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_platform_suggestions.py +++ /dev/null @@ -1,163 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element -* Part of the video platform prototype -* Creates a list of video suggestions -* Based on word similarity ranking -* Must be done in a collection that holds video records with thumbnails, title and author -""" - -from invenio.config import CFG_BASE_URL -from invenio.legacy.bibdocfile.api import BibRecDocs -from intbitset import intbitset -from invenio.legacy.search_engine import perform_request_search -from invenio.legacy.bibrank.record_sorter import rank_records -from invenio.legacy.bibrecord import get_fieldvalues -from invenio.modules.encoder.utils import timecode_to_seconds -import random - -html_skeleton_suggestion = """ - -
    -
    - - %(video_thumb_alt)s - -
    - %(video_duration)s -
    -
    -
    - %(video_title)s -
    -
    - by %(video_authors)s -
    -
    -""" - -def format_element(bfo, collection="Videos", threshold="75", maximum="3", shuffle="True"): - """ Creates video suggestions based on ranking algorithms - - @param collection: Collection to take the suggestions from - @param threshold: Value between 0 and 100. Only records ranked higher than the value are presented. - @param maximum: Maximum suggestions to show - @param shuffle: True or False, should the suggestions be shuffled? - """ - if threshold.isdigit(): - threshold = int(threshold) - else: - raise ValueError("The given threshold is not a digit") - if maximum.isdigit(): - maximum = int(maximum) - else: - raise ValueError("The given maximum is not a digit") - if shuffle == "True": - shuffle = True - else: - shuffle = False; - suggestions = [] - recid = bfo.control_field('001') - similar_records = find_similar_videos(recid, collection, threshold, maximum, shuffle) - for sim_recid in similar_records: - thumbnail = get_video_thumbnail(sim_recid) - title = get_video_title(sim_recid) - authors = get_video_authors(sim_recid) - url = get_video_record_url(sim_recid) - duration = get_video_duration(sim_recid) - suggestion = html_skeleton_suggestion % { - 'video_record_url': url, - 'video_thumb_url': thumbnail[0], - 'video_thumb_alt': thumbnail[1], - 'video_duration': duration, - 'video_title': title, - 'video_authors': authors, - } - suggestions.append(suggestion) - return "\n".join(suggestions) - -def find_similar_videos(recid, collection="Videos", threshold=75, maximum=3, shuffle=True): - """ Returns a list of similar video records - """ - similar_records = [] - collection_recids = intbitset(perform_request_search(cc=collection)) - ranking = rank_records('wrd', 0, collection_recids, ['recid:' + str(recid)]) - ## ([6, 7], [81, 100], '(', ')', '') - for list_pos, rank in enumerate(ranking[1]): - if rank >= threshold: - similar_records.append(ranking[0][list_pos]) - if shuffle: - if maximum > len(similar_records): - maximum = len(similar_records) - return random.sample(similar_records, maximum) - else: - return similar_records[:maximum] - -def get_video_thumbnail(recid): - """ Returns the URL and ALT text for a video thumbnail of a given record - """ - comments = get_fieldvalues(recid, '8564_z') - descriptions = get_fieldvalues(recid, '8564_y') - urls = get_fieldvalues(recid, '8564_u') - for pos, comment in enumerate(comments): - if comment in ('SUGGESTIONTHUMB', 'BIGTHUMB', 'THUMB', 'SMALLTHUMB', 'POSTER'): - return (urls[pos], descriptions[pos]) - return ("", "") - -def get_video_title(recid): - """ Return the Title of a video record - """ - return get_fieldvalues(recid, '245__a')[0] - -def get_video_authors(recid): - """ Return the Authors of a video record - """ - return ", ".join(get_fieldvalues(recid, '100__a')) - -def get_video_record_url(recid): - """ Return the URL of a video record - """ - return CFG_BASE_URL + "/record/" + str(recid) - -def get_video_duration(recid): - """ Return the duration of a video - """ - duration = get_fieldvalues(recid, '950__d') - if duration: - duration = duration[0] - duration = timecode_to_seconds(duration) - return human_readable_time(duration) - else: - return "" - -def human_readable_time(seconds): - """ Creates a human readable duration representation - """ - for x in ['s','m','h']: - if seconds < 60.0: - return "%.0f %s" % (seconds, x) - seconds /= seconds - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_video_selector.py b/invenio/modules/formatter/format_elements/bfe_video_selector.py deleted file mode 100644 index 4275f23e7..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_selector.py +++ /dev/null @@ -1,109 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Creates a - %s - - """ % '\n'.join(options) - return text - - -def create_option_element(width, height, subformat, url_webm=None, url_ogv=None, url_mp4=None, url_poster=None,): - """ Creates an HTML option element that carries all video information - """ - if url_webm: - webm = """data-src-webm="%s" data-type-webm='video/webm; codecs="vp8, vorbis"'""" % url_webm - else: - webm = "" - if url_ogv: - ogv = """data-src-ogg="%s" data-type-ogv='video/ogv; codecs="theora, vorbis"'""" % url_ogv - else: - ogv = "" - if url_mp4: - mp4 = """data-src-mp4="%s" data-type-mp4='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'""" % url_mp4 - else: - mp4 = "" - text = """""" % { - 'webm': webm, - 'ogv': ogv, - 'mp4': mp4, - 'url_poster': url_poster, - 'width': width, - 'height': height, - 'subformat': subformat - } - return text - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_video_sources.py b/invenio/modules/formatter/format_elements/bfe_video_sources.py deleted file mode 100644 index 04e085765..000000000 --- a/invenio/modules/formatter/format_elements/bfe_video_sources.py +++ /dev/null @@ -1,57 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Creates elements for html5 videos -""" - -from invenio.legacy.bibdocfile.api import BibRecDocs - -def format_element(bfo, subformat="480p"): - """ Creates HTML5 source elements for the given subformat. - - MP4, WebM and OGV are currently supported as video sources. - The function will scan the bibdocfiles attached to the record for - videos with these formats and the fiven subformat. - - @param subformat: BibDocFile subformat to create the sources from (e.g. 480p) - """ - video_sources = [] - recdoc = BibRecDocs(bfo.recID) - bibdocs = recdoc.list_bibdocs() - for bibdoc in bibdocs: - bibdocfiles = bibdoc.list_all_files() - for bibdocfile in bibdocfiles: - if bibdocfile.get_superformat() in ('.mp4', '.webm', '.ogv') and bibdocfile.get_subformat() == subformat: - src = bibdocfile.get_url() - ftype = bibdocfile.get_superformat()[1:] - if ftype == 'mp4': - codecs = 'avc1.42E01E, mp4a.40.2' - elif ftype == 'webm': - codecs = 'vp8, vorbis' - elif ftype == 'ogv': - codecs = 'theora, vorbis' - source = '' % (src, ftype, codecs) - video_sources.append(source) - return "\n".join(video_sources) - -def escape_values(bfo): - """ - Called by BibFormat in order to check if output of this element - should be escaped. - """ - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_xml_record.py b/invenio/modules/formatter/format_elements/bfe_xml_record.py deleted file mode 100644 index 3140cd957..000000000 --- a/invenio/modules/formatter/format_elements/bfe_xml_record.py +++ /dev/null @@ -1,44 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""BibFormat element - Prints record as XML.""" - - -def format_element(bfo, type='xm', encodeForXML='yes'): - """Print the complete current record as XML. - - :param type: the type of xml. Can be 'xml', 'oai_dc', 'marcxml', 'xd' - :param encodeForXML: if 'yes', replace all < > and & with html - corresponding escaped characters. - """ - assert type == 'xm' - from invenio.modules.records.api import get_record - from invenio.utils.text import encode_for_xml - - out = get_record(bfo.recID).legacy_export_as_marc() - - if encodeForXML.lower() == 'yes': - return encode_for_xml(out) - else: - return out - - -def escape_values(bfo): - """Check if output of this element should be escaped.""" - return 0 diff --git a/invenio/modules/formatter/format_elements/bfe_year.py b/invenio/modules/formatter/format_elements/bfe_year.py deleted file mode 100644 index 3096c2d48..000000000 --- a/invenio/modules/formatter/format_elements/bfe_year.py +++ /dev/null @@ -1,34 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the publication year -""" -__revision__ = "$Id$" - -import re - -def format_element(bfo): - """ - Prints the publication year. - @see: pagination.py, publisher.py, reprints.py, imprint.py, place.py - """ - for date_field in ['773__y', '260__c', '269__c', '909C4y', '925__a']: - date = bfo.field(date_field) - match_obj = re.search('\d\d\d\d', date) - if match_obj is not None: - return match_obj.group() diff --git a/invenio/modules/formatter/manage.py b/invenio/modules/formatter/manage.py index 7ac778251..f17ad2492 100644 --- a/invenio/modules/formatter/manage.py +++ b/invenio/modules/formatter/manage.py @@ -1,270 +1,70 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. -# Copyright (C) 2013, 2014 CERN. +# Copyright (C) 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """ Perform template migration operations. Migrate output formats and output templates found in ``CFG_BIBFORMAT_OUTPUTS_PATH`` and ``CFG_BIBFORMAT_TEMPLATES_PATH`` respectively. It creates backup of each output format with name ``_legacy.bfo`` and generates new Jinja2 templates in ``CFG_BIBFORMAT_JINJA_TEMPLATE_PATH``. """ from __future__ import print_function import os import re import shutil from six import iteritems from invenio.ext.script import Manager manager = Manager(usage="Perform template migration operations.") -@manager.option('--rewrite-existing-templates', - dest='rewrite_existing_templates', - action='store_true', default=False) -@manager.option('-t', '--template', - dest='only_template_re', default=None, - help="only templates matching regular expression") -@manager.option('--verbose', dest='verbose') -def bft2tpl(rewrite_existing_templates=False, only_template_re=None, - verbose=0): - """Convert *bft* templates to Jinja2 *tpl* templates.""" - # Import all invenio modules inside to avoid side-efects ouside - # Flask application context. - from invenio.modules.formatter.config import CFG_BIBFORMAT_OUTPUTS_PATH, \ - CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION, \ - CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, \ - CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION, \ - CFG_BIBFORMAT_JINJA_TEMPLATE_PATH - from invenio.modules.formatter.engine import get_format_element, \ - get_output_formats, \ - pattern_function_params, \ - pattern_tag, pattern_lang, \ - translation_pattern, \ - ln_pattern, get_format_templates - from invenio.legacy.bibformat.adminlib import \ - update_output_format_rules - - only_template = re.compile(only_template_re) \ - if only_template_re is not None else None - - def rename_template(template): - if template[-3:] == CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION and \ - (only_template is None or only_template.match(template)): - return template[:-3] + \ - CFG_BIBFORMAT_FORMAT_JINJA_TEMPLATE_EXTENSION - return template - - def update_rule(rule): - rule['template'] = rename_template(rule['template']) - print(' ...', rule['template'], 'to', end=' ') - print(rename_template(rule['template'])) - print(' ', rule) - return rule - - def eval_format_template_elements(format_template, bfo, verbose=0): - - def insert_element_code(match): - error = [] - function_name = match.group("function_name") - try: - format_element = get_format_element(function_name, verbose) - except Exception: - error.append('Invalid function name %s' % (function_name, )) - - params_str = [] - if format_element is not None: - params = {} - # Look for function parameters given in format template code - all_params = match.group('params') - if all_params is not None: - function_params_iterator = pattern_function_params.\ - finditer(all_params) - for param_match in function_params_iterator: - sep = param_match.group('sep') - name = param_match.group('param') - value = param_match.group('value') - params[name] = value - params_str.append(name + '=' + sep + value + sep) - - # Replace element with function call with params. - result = '{{ bfe_%s(bfo, %s) }}' % (function_name.lower(), - ', '.join(params_str)) - return result - - print('\n'.join(error)) - - # Substitute special tags in the format by our own text. - # Special tags have the form - # - format = pattern_tag.sub(insert_element_code, format_template) - return format - - def translate(match): - """Translate matching values.""" - word = match.group("word") - translated_word = '{{ _("' + word + '") }}' - return translated_word - - def filter_languages(format_template): - """Filter languages in format template.""" - def search_lang_tag(match): - """Searche for the ... tag.""" - ln_tags = {} - - def clean_language_tag(match): - """Return tag text content. - - It contains if statement block to match output language. - It is called by substitution in 'filter_languages(...)'. - - @param match: a match object corresponding to the special tag - that must be interpreted - """ - ln_tags[match.group(1)] = match.group(2) - return '{% if g.ln == "' + match.group(1) + '" %}' + \ - match.group(2) + '{% endif %}' - - # End of clean_language_tag - - lang_tag_content = match.group("langs") - return '{% lang %}' + lang_tag_content + '{% endlang %}' - cleaned_lang_tag = ln_pattern.sub(clean_language_tag, - lang_tag_content) - # FIXME no traslation for current language - # if len(ln_tags) > 0: - # cleaned_lang_tag += '{% if not g.ln in ["' + \ - # '", "'.join(ln_tags.keys()) + '"] %}' + \ - # ln_tags.get(CFG_SITE_LANG, '') + '{% endif %}' - return cleaned_lang_tag - # End of search_lang_tag - - filtered_format_template = pattern_lang.sub(search_lang_tag, - format_template) - return filtered_format_template - - skip_templates = lambda (name, key): name[-3:] != 'xsl' - format_templates = filter(skip_templates, - iteritems(get_format_templates(True))) - - print('>>> Going to migrate %d format template(s) ...' % ( - len(format_templates), )) - - if not os.path.exists(CFG_BIBFORMAT_JINJA_TEMPLATE_PATH): - os.makedirs(CFG_BIBFORMAT_JINJA_TEMPLATE_PATH) - - for name, template in format_templates: - - if not (only_template is None or only_template.match(name)): - continue - - new_name = os.path.join(CFG_BIBFORMAT_JINJA_TEMPLATE_PATH, - rename_template(name)) - - if os.path.exists(new_name): - print(' [!] File', new_name, 'already exists.', end=' ') - if not rewrite_existing_templates: - print('Skipped.') - continue - else: - shutil.copy2(new_name, new_name + '.backup') - print('Rewritten.') - - print(' ... migrating', name, 'to', new_name) - - with open(new_name, 'w+') as f: - code = template['code'] - ln_tags_format = filter_languages(code) - localized_format = translation_pattern.sub(translate, - ln_tags_format) - evaled = eval_format_template_elements(localized_format, None) - f.write(evaled) - - print() - - skip_legacy = lambda (name, key): name[-11:] != '_legacy.' + \ - CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION - output_formats = filter( - skip_legacy, iteritems(get_output_formats(with_attributes=True))) - print('>>> Going to migrate %d output format(s) ...' % ( - len(output_formats))) - - for name, output_format in output_formats: - if not any(map(lambda rule: rule['template'][-3:] == - CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, - output_format['rules'])): - print(' [!]', name, 'does not contain any', end=' ') - print(CFG_BIBFORMAT_FORMAT_TEMPLATE_EXTENSION, 'template', end=' ') - if only_template is not None: - print('or does not match', only_template_re, end=' ') - print('.') - continue - - new_name = name[:-4] + \ - '_legacy.' + CFG_BIBFORMAT_FORMAT_OUTPUT_EXTENSION - if os.path.exists(os.path.join(CFG_BIBFORMAT_OUTPUTS_PATH, new_name)): - print(' [!] File', new_name, 'already exists. Skipped.') - continue - shutil.copy2( - os.path.join(CFG_BIBFORMAT_OUTPUTS_PATH, name), - os.path.join(CFG_BIBFORMAT_OUTPUTS_PATH, new_name)) - # rename template names - print(' ... migrating', name, 'to', new_name) - update_output_format_rules(name, - map(update_rule, output_format['rules']), - rename_template(output_format['default'])) - - print() - print('>>> Please re-run `bibreformat` for all cached output formats.') - print(' $ bibreformat -oHB,HD -a') - - @manager.option('-o', '--output-format', dest='output_format', default="HB", help="Specify output format/s (default HB)") def expunge(output_format="HB"): """Remove static output formats from cache.""" from invenio.ext.sqlalchemy import db from invenio.modules.formatter.models import Bibfmt # Make it uppercased as it is stored in database. output_format = output_format.upper() print(">>> Cleaning %s cache..." % (output_format, )) # Prepare where expression. filter_format = ( Bibfmt.format == output_format if ',' not in output_format else Bibfmt.format.in_(map(lambda x: x.strip(), output_format.split(','))) ) Bibfmt.query.filter(filter_format).delete(synchronize_session=False) db.session.commit() def main(): """Run manager.""" from invenio.base.factory import create_app app = create_app() manager.app = app manager.run() if __name__ == '__main__': main() diff --git a/invenio/modules/formatter/registry.py b/invenio/modules/formatter/registry.py index 3c77391a8..4e020cc39 100644 --- a/invenio/modules/formatter/registry.py +++ b/invenio/modules/formatter/registry.py @@ -1,119 +1,113 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA """Implement registries for formatter.""" import os from flask_registry import ( ModuleAutoDiscoveryRegistry, PkgResourcesDirDiscoveryRegistry, RegistryProxy, ) from invenio.ext.registry import ModuleAutoDiscoverySubRegistry from invenio.utils.datastructures import LazyDict import yaml -format_elements = RegistryProxy( - 'format_elements', - ModuleAutoDiscoverySubRegistry, - 'format_elements' -) - format_templates_directories = RegistryProxy( 'format_templates_directories', ModuleAutoDiscoveryRegistry, 'format_templates' ) format_templates = RegistryProxy( 'format_templates', PkgResourcesDirDiscoveryRegistry, '.', registry_namespace=format_templates_directories ) output_formats_directories = RegistryProxy( 'output_formats_directories', ModuleAutoDiscoveryRegistry, 'output_formats' ) output_formats_files = RegistryProxy( 'output_formats_files', PkgResourcesDirDiscoveryRegistry, '.', registry_namespace=output_formats_directories ) template_context_functions = RegistryProxy( 'template_context_functions', ModuleAutoDiscoverySubRegistry, 'template_context_functions' ) def create_format_templates_lookup(): """Create format templates.""" out = {} def _register(path, level=1): if level > 4: return normpath = os.path.normpath(path) if os.path.isdir(normpath): for p in os.listdir(normpath): _register(os.path.join(normpath, p), level=level+1) else: parts = normpath.split(os.path.sep) out[os.path.sep.join(parts[-level:])] = normpath for t in reversed(format_templates): _register(t) return out format_templates_lookup = LazyDict(create_format_templates_lookup) def create_output_formats_lookup(): """Create output formats.""" out = {} for f in output_formats_files: of = os.path.basename(f).lower() data = {'names': {}} if of.endswith('.yml'): of = of[:-4] with open(f, 'r') as f: data.update(yaml.load(f) or {}) data['code'] = of else: continue # unknown filetype if of in out: continue out[of] = data return out output_formats = LazyDict(create_output_formats_lookup) export_formats = LazyDict(lambda: dict( (code, of) for code, of in output_formats.items() if of.get('content_type', '') != 'text/html' and of.get('visibility', 0) )) diff --git a/invenio/modules/formatter/testsuite/format_elements/__init__.py b/invenio/modules/formatter/testsuite/format_elements/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/invenio/modules/formatter/testsuite/format_elements/bfe_test_2.py b/invenio/modules/formatter/testsuite/format_elements/bfe_test_2.py deleted file mode 100644 index e6efd9b12..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/bfe_test_2.py +++ /dev/null @@ -1,32 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - test element that has no docstring. -""" -__revision__ = "$Id$" - -def format_element(bfo, param1, param2='default value'): - """ - Prints test - - @param param1: desc 1 - @param param2: desc 2 - @param unknown_param: desc3 - @see: element2.py unknown_element.py - """ - return "test" diff --git a/invenio/modules/formatter/testsuite/format_elements/bfe_test_4.py b/invenio/modules/formatter/testsuite/format_elements/bfe_test_4.py deleted file mode 100644 index 2f4b10900..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/bfe_test_4.py +++ /dev/null @@ -1,34 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -__revision__ = "$Id$" - - -def format_element(bfo, param1, param2='default value'): - """ - Prints test - - @param param1: desc 1 - @param param2: desc 2 - @param unknown_param: desc3 - @see: element2.py unknown_element.py - """ - # Deliberate broken element used for testing. - unloadable element - return "test" diff --git a/invenio/modules/formatter/testsuite/format_elements/bfe_test_6.py b/invenio/modules/formatter/testsuite/format_elements/bfe_test_6.py deleted file mode 100644 index c15561b56..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/bfe_test_6.py +++ /dev/null @@ -1,25 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2013 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - - -def format_element(bfo): - """ - Prints a tag - """ - return "helloworld" diff --git a/invenio/modules/formatter/testsuite/format_elements/test3.py b/invenio/modules/formatter/testsuite/format_elements/test3.py deleted file mode 100644 index 8b1378917..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/test3.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/invenio/modules/formatter/testsuite/format_elements/test_1.py b/invenio/modules/formatter/testsuite/format_elements/test_1.py deleted file mode 100644 index 1e3233ba3..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/test_1.py +++ /dev/null @@ -1,34 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - test file with unknown param docstring and reference -to unknown element -""" -__revision__ = "$Id$" - -def format_element(bfo, param1, param2='default value'): - """ - Prints test - - @param param1: desc 1 - @param param2: desc 2 - @param unknown_param: desc3 - @see: element2.py, unknown_element.py - """ - - return "test" diff --git a/invenio/modules/formatter/testsuite/format_elements/test_5.py b/invenio/modules/formatter/testsuite/format_elements/test_5.py deleted file mode 100644 index 6eb814a7e..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/test_5.py +++ /dev/null @@ -1,28 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2010, 2011 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - test file with unknown param docstring and reference -to unknown element -""" -__revision__ = "$Id$" - -def format_element(bfo, param1, param2, param3, param4="a default value"): - - #No docstring on purpose - - return param1 + param2 + param3 + param4 + bfo.field("100.b") diff --git a/invenio/modules/formatter/testsuite/format_elements/test_no_element.test b/invenio/modules/formatter/testsuite/format_elements/test_no_element.test deleted file mode 100644 index 8d1c8b69c..000000000 --- a/invenio/modules/formatter/testsuite/format_elements/test_no_element.test +++ /dev/null @@ -1 +0,0 @@ - diff --git a/invenio/modules/formatter/testsuite/format_templates/Test3.bft b/invenio/modules/formatter/testsuite/format_templates/Test3.bft deleted file mode 100644 index 1e7bd6e72..000000000 --- a/invenio/modules/formatter/testsuite/format_templates/Test3.bft +++ /dev/null @@ -1,8 +0,0 @@ -test -test -

    hi

    this is my template -tentfrgarbage - - - - \ No newline at end of file diff --git a/invenio/modules/formatter/testsuite/format_templates/Test6.bft b/invenio/modules/formatter/testsuite/format_templates/Test6.bft deleted file mode 100644 index 9d137a945..000000000 --- a/invenio/modules/formatter/testsuite/format_templates/Test6.bft +++ /dev/null @@ -1 +0,0 @@ - diff --git a/invenio/modules/formatter/testsuite/format_templates/Test_no_template.test b/invenio/modules/formatter/testsuite/format_templates/Test_no_template.test deleted file mode 100644 index 8d1c8b69c..000000000 --- a/invenio/modules/formatter/testsuite/format_templates/Test_no_template.test +++ /dev/null @@ -1 +0,0 @@ - diff --git a/invenio/modules/formatter/testsuite/format_templates/__init__.py b/invenio/modules/formatter/testsuite/format_templates/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/invenio/modules/formatter/testsuite/output_formats/test3.yml b/invenio/modules/formatter/testsuite/output_formats/test3.yml index 6569bc3e3..b739681c3 100644 --- a/invenio/modules/formatter/testsuite/output_formats/test3.yml +++ b/invenio/modules/formatter/testsuite/output_formats/test3.yml @@ -1,4 +1,4 @@ rules: -- {field: 980.a, template: Unknown.bft, value: 'PICTURE'} -- {field: 980.a, template: Test3.bft, value: 'ARTICLE'} -default: Default_unknown.bft +- {field: 980.a, template: Unknown.tpl, value: 'PICTURE'} +- {field: 980.a, template: Test3.tpl, value: 'ARTICLE'} +default: Default_unknown.tpl diff --git a/invenio/modules/formatter/testsuite/output_formats/test6.yml b/invenio/modules/formatter/testsuite/output_formats/test6.yml index 8106a2dd2..a3f873667 100644 --- a/invenio/modules/formatter/testsuite/output_formats/test6.yml +++ b/invenio/modules/formatter/testsuite/output_formats/test6.yml @@ -1 +1 @@ -default: Test6.bft +default: Test6.tpl diff --git a/invenio/modules/formatter/testsuite/output_formats/test7.yml b/invenio/modules/formatter/testsuite/output_formats/test7.yml index 4c3f45f34..1cb2831b2 100644 --- a/invenio/modules/formatter/testsuite/output_formats/test7.yml +++ b/invenio/modules/formatter/testsuite/output_formats/test7.yml @@ -1 +1 @@ -default: Test7.bft +default: Test7.tpl diff --git a/invenio/modules/formatter/testsuite/output_formats/test8.yml b/invenio/modules/formatter/testsuite/output_formats/test8.yml index 3da63c1c2..21d070cd2 100644 --- a/invenio/modules/formatter/testsuite/output_formats/test8.yml +++ b/invenio/modules/formatter/testsuite/output_formats/test8.yml @@ -1 +1 @@ -default: Test8.bft +default: Test8.tpl diff --git a/invenio/modules/formatter/testsuite/format_templates/Test1.bft b/invenio/modules/formatter/testsuite/templates/format/record/Test1.tpl similarity index 100% rename from invenio/modules/formatter/testsuite/format_templates/Test1.bft rename to invenio/modules/formatter/testsuite/templates/format/record/Test1.tpl diff --git a/invenio/modules/formatter/testsuite/templates/format/record/Test3.tpl b/invenio/modules/formatter/testsuite/templates/format/record/Test3.tpl new file mode 100644 index 000000000..8bdb4a8ca --- /dev/null +++ b/invenio/modules/formatter/testsuite/templates/format/record/Test3.tpl @@ -0,0 +1 @@ +

    hi

    this is my template diff --git a/invenio/modules/formatter/testsuite/templates/format/record/Test6.tpl b/invenio/modules/formatter/testsuite/templates/format/record/Test6.tpl new file mode 100644 index 000000000..d35563e8c --- /dev/null +++ b/invenio/modules/formatter/testsuite/templates/format/record/Test6.tpl @@ -0,0 +1,2 @@ +helloworld + diff --git a/invenio/modules/formatter/testsuite/format_templates/Test7.bft b/invenio/modules/formatter/testsuite/templates/format/record/Test7.tpl similarity index 100% rename from invenio/modules/formatter/testsuite/format_templates/Test7.bft rename to invenio/modules/formatter/testsuite/templates/format/record/Test7.tpl diff --git a/invenio/modules/formatter/testsuite/format_templates/Test8.bft b/invenio/modules/formatter/testsuite/templates/format/record/Test8.tpl similarity index 100% rename from invenio/modules/formatter/testsuite/format_templates/Test8.bft rename to invenio/modules/formatter/testsuite/templates/format/record/Test8.tpl diff --git a/invenio/modules/formatter/testsuite/format_templates/Test_2.bft b/invenio/modules/formatter/testsuite/templates/format/record/Test_2.tpl similarity index 100% rename from invenio/modules/formatter/testsuite/format_templates/Test_2.bft rename to invenio/modules/formatter/testsuite/templates/format/record/Test_2.tpl diff --git a/invenio/modules/formatter/testsuite/test_formatter_engine.py b/invenio/modules/formatter/testsuite/test_formatter_engine.py index be2bc6d69..f03d1d8af 100644 --- a/invenio/modules/formatter/testsuite/test_formatter_engine.py +++ b/invenio/modules/formatter/testsuite/test_formatter_engine.py @@ -1,980 +1,665 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Test cases for the BibFormat engine. Also test some utilities function in bibformat_utils module""" __revision__ = "$Id$" # pylint: disable=C0301 -import os import pkg_resources import sys +from invenio.base.config import PACKAGES from invenio.base.globals import cfg from invenio.base.wrappers import lazy_import from invenio.testsuite import make_test_suite, run_test_suite, InvenioTestCase -from invenio.ext.registry import ModuleAutoDiscoverySubRegistry from flask_registry import PkgResourcesDirDiscoveryRegistry, RegistryProxy, \ ImportPathRegistry, ModuleAutoDiscoveryRegistry bibformat = lazy_import('invenio.modules.formatter') bibformat_engine = lazy_import('invenio.modules.formatter.engine') bibformat_utils = lazy_import('invenio.modules.formatter.utils') bibformat_config = lazy_import('invenio.modules.formatter.config') -bibformatadminlib = lazy_import('invenio.legacy.bibformat.adminlib') -format_templates = lazy_import('invenio.modules.formatter.testsuite.format_templates') gettext_set_language = lazy_import('invenio.base.i18n:gettext_set_language') TEST_PACKAGES = [ 'invenio.modules.formatter.testsuite.overlay', 'invenio.modules.formatter.testsuite', ] test_registry = RegistryProxy('test_registry', ImportPathRegistry, initial=TEST_PACKAGES) -format_templates_registry = lambda: PkgResourcesDirDiscoveryRegistry( - 'format_templates', registry_namespace=test_registry) - -format_elements_registry = lambda: ModuleAutoDiscoverySubRegistry( - 'format_elements', registry_namespace=test_registry, silent=True) - output_formats_directories_registry = lambda: ModuleAutoDiscoveryRegistry( 'output_formats', registry_namespace=test_registry, silent=True ) -class FormatTemplateTest(InvenioTestCase): - """ bibformat - tests on format templates""" - - def setUp(self): - self.app.extensions['registry']['format_templates'] = format_templates_registry() - - def tearDown(self): - del self.app.extensions['registry']['format_templates'] - - def test_get_format_template(self): - """bibformat - format template parsing and returned structure""" - - #Test correct parsing and structure - template_1 = bibformat_engine.get_format_template("Test1.bft", with_attributes=True) - self.assert_(template_1 is not None) - self.assertEqual(template_1['code'], "test\nthis value should stay as it is\nthis one too\n") - self.assertEqual(template_1['attrs']['name'], "name_test") - self.assertEqual(template_1['attrs']['description'], "desc_test") - - #Test correct parsing and structure of file without description or name - template_2 = bibformat_engine.get_format_template("Test_2.bft", with_attributes=True) - self.assert_(template_2 is not None) - self.assertEqual(template_2['code'], "test") - self.assertEqual(template_2['attrs']['name'], "Test_2.bft") - self.assertEqual(template_2['attrs']['description'], "") - - #Test correct parsing and structure of file without description or name - unknown_template = bibformat_engine.get_format_template("test_no_template.test", with_attributes=True) - self.assertEqual(unknown_template, None) - - - def test_get_format_templates(self): - """ bibformat - loading multiple format templates""" - templates = bibformat_engine.get_format_templates(with_attributes=True) - #test correct loading - self.assert_("Test1.bft" in templates.keys()) - self.assert_("Test_2.bft" in templates.keys()) - self.assert_("Test3.bft" in templates.keys()) - self.assert_("Test_no_template.test" not in templates.keys()) - - #Test correct pasrsing and structure - self.assertEqual(templates['Test1.bft']['code'], "test\nthis value should stay as it is\nthis one too\n") - self.assertEqual(templates['Test1.bft']['attrs']['name'], "name_test") - self.assertEqual(templates['Test1.bft']['attrs']['description'], "desc_test") - - def test_get_format_template_attrs(self): - """ bibformat - correct parsing of attributes in format template""" - attrs = bibformat_engine.get_format_template_attrs("Test1.bft") - self.assertEqual(attrs['name'], "name_test") - self.assertEqual(attrs['description'], "desc_test") - - -class FormatElementTest(InvenioTestCase): - """ bibformat - tests on format templates""" - - def setUp(self): - # pylint: disable=C0103 - """bibformat - setting python path to test elements""" - self.app.extensions['registry']['format_elements'] = format_elements_registry() - - def tearDown(self): - del self.app.extensions['registry']['format_elements'] - - def test_resolve_format_element_filename(self): - """bibformat - resolving format elements filename """ - #Test elements filename starting without bfe_, with underscore instead of space - filenames = ["test 1", "test 1.py", "bfe_test 1", "bfe_test 1.py", "BFE_test 1", - "BFE_TEST 1", "BFE_TEST 1.py", "BFE_TeST 1.py", "BFE_TeST 1", - "BfE_TeST 1.py", "BfE_TeST 1", "test_1", "test_1.py", "bfe_test_1", - "bfe_test_1.py", "BFE_test_1", - "BFE_TEST_1", "BFE_TEST_1.py", "BFE_Test_1.py", "BFE_TeST_1", - "BfE_TeST_1.py", "BfE_TeST_1"] - - for i in range(len(filenames)-2): - filename_1 = bibformat_engine.resolve_format_element_filename(filenames[i]) - self.assert_(filename_1 is not None) - - filename_2 = bibformat_engine.resolve_format_element_filename(filenames[i+1]) - self.assertEqual(filename_1, filename_2) - - - #Test elements filename starting with bfe_, and with underscores instead of spaces - filenames = ["test 2", "test 2.py", "bfe_test 2", "bfe_test 2.py", "BFE_test 2", - "BFE_TEST 2", "BFE_TEST 2.py", "BFE_TeST 2.py", "BFE_TeST 2", - "BfE_TeST 2.py", "BfE_TeST 2", "test_2", "test_2.py", "bfe_test_2", - "bfe_test_2.py", "BFE_test_2", - "BFE_TEST_2", "BFE_TEST_2.py", "BFE_TeST_2.py", "BFE_TeST_2", - "BfE_TeST_2.py", "BfE_TeST_2"] - - for i in range(len(filenames)-2): - filename_1 = bibformat_engine.resolve_format_element_filename(filenames[i]) - self.assert_(filename_1 is not None) - - filename_2 = bibformat_engine.resolve_format_element_filename(filenames[i+1]) - self.assertEqual(filename_1, filename_2) - - #Test non existing element - non_existing_element = bibformat_engine.resolve_format_element_filename("BFE_NON_EXISTING_ELEMENT") - self.assertEqual(non_existing_element, None) - - def test_get_format_element(self): - """bibformat - format elements parsing and returned structure""" - #Test loading with different kind of names, for element with spaces in name, without bfe_ - element_1 = bibformat_engine.get_format_element("test 1", with_built_in_params=True) - self.assert_(element_1 is not None) - element_1_bis = bibformat_engine.get_format_element("bfe_tEst_1.py", with_built_in_params=True) - self.assertEqual(element_1, element_1_bis) - - #Test loading with different kind of names, for element without spaces in name, wit bfe_ - element_2 = bibformat_engine.get_format_element("test 2", with_built_in_params=True) - self.assert_(element_2 is not None) - element_2_bis = bibformat_engine.get_format_element("bfe_tEst_2.py", with_built_in_params=True) - self.assertEqual(element_2, element_2_bis) - - #Test loading incorrect elements - try: - element_3 = bibformat_engine.get_format_element("test 3", with_built_in_params=True) - except bibformat_engine.InvenioBibFormatError as e: - self.assertEqual(str(e), 'Format element test 3 could not be found.') - else: - self.fail("Should have raised InvenioBibFormatError") - - try: - element_4 = bibformat_engine.get_format_element("test 4", with_built_in_params=True) - except bibformat_engine.InvenioBibFormatError as e: - self.assertEqual(str(e), 'Format element test 4 could not be found.') - else: - self.fail("Should have raised SyntaxError") - - try: - unknown_element = bibformat_engine.get_format_element("TEST_NO_ELEMENT", with_built_in_params=True) - except bibformat_engine.InvenioBibFormatError as e: - self.assertEqual(str(e), 'Format element TEST_NO_ELEMENT could not be found.') - else: - self.fail("Should have raised InvenioBibFormatError") - - #Test element without docstring - element_5 = bibformat_engine.get_format_element("test_5", with_built_in_params=True) - self.assert_(element_5 is not None) - self.assertEqual(element_5['attrs']['description'], '') - self.assert_({'name': "param1", - 'description': "(no description provided)", - 'default': ""} in element_5['attrs']['params']) - self.assertEqual(element_5['attrs']['seealso'], []) - - #Test correct parsing: - - #Test type of element - self.assertEqual(element_1['type'], "python") - #Test name = element filename, with underscore instead of spaces, - #without BFE_ and uppercase - self.assertEqual(element_1['attrs']['name'], "TEST_1") - #Test description parsing - self.assertEqual(element_1['attrs']['description'], "Prints test") - #Test @see: parsing - self.assertEqual(element_1['attrs']['seealso'], ["element2.py", "unknown_element.py"]) - #Test @param parsing - self.assert_({'name': "param1", - 'description': "desc 1", - 'default': ""} in element_1['attrs']['params']) - - self.assert_({'name': "param2", - 'description': "desc 2", - 'default': "default value"} in element_1['attrs']['params']) - - - - #Test non existing element - try: - non_existing_element = bibformat_engine.get_format_element("BFE_NON_EXISTING_ELEMENT") - except bibformat_engine.InvenioBibFormatError as e: - self.assertEqual(str(e), 'Format element BFE_NON_EXISTING_ELEMENT could not be found.') - else: - self.fail("Should have raised InvenioBibFormatError") - - def test_get_format_element_attrs_from_function(self): - """ bibformat - correct parsing of attributes in 'format' docstring""" - element_1 = bibformat_engine.get_format_element("test 1", with_built_in_params=True) - function = element_1['code'] - attrs = bibformat_engine.get_format_element_attrs_from_function(function, - element_1['attrs']['name'], - with_built_in_params=True) - - self.assertEqual(attrs['name'], "TEST_1") - #Test description parsing - self.assertEqual(attrs['description'], "Prints test") - #Test @see: parsing - self.assertEqual(attrs['seealso'], ["element2.py", "unknown_element.py"]) - - def test_get_format_elements(self): - """bibformat - multiple format elements parsing and returned structure""" - elements = bibformat_engine.get_format_elements() - self.assert_(isinstance(elements, dict)) - self.assertEqual(elements['TEST_1']['attrs']['name'], "TEST_1") - self.assertEqual(elements['TEST_2']['attrs']['name'], "TEST_2") - self.assert_("TEST_3" not in elements.keys()) - self.assert_("TEST_4" not in elements.keys()) - - def test_get_tags_used_by_element(self): - """bibformat - identification of tag usage inside element""" - del self.app.extensions['registry']['format_elements'] - from invenio.modules.formatter.registry import format_elements - list(format_elements) - bibformat_engine.TEMPLATE_CONTEXT_FUNCTIONS_CACHE.bibformat_elements.cache.clear() - #cfg['CFG_BIBFORMAT_ELEMENTS_IMPORT_PATH'] = self.old_import_path - tags = bibformatadminlib.get_tags_used_by_element('bfe_abstract.py') - self.failUnless(len(tags) == 4, - 'Could not correctly identify tags used in bfe_abstract.py') - class OutputFormatTest(InvenioTestCase): """ bibformat - tests on output formats""" def setUp(self): self.app.extensions['registry']['output_formats_directories'] = \ output_formats_directories_registry() from invenio.modules.formatter.registry import output_formats as ofs ofs.expunge() def tearDown(self): from invenio.modules.formatter.registry import output_formats as ofs ofs.expunge() del self.app.extensions['registry']['output_formats_directories'] def test_get_output_format(self): """ bibformat - output format parsing and returned structure """ from invenio.modules.formatter.registry import output_formats as ofs output_1 = ofs['test1'] #self.assertEqual(output_1['attrs']['names']['generic'], "") #self.assert_(isinstance(output_1['attrs']['names']['ln'], dict)) #self.assert_(isinstance(output_1['attrs']['names']['sn'], dict)) self.assertEqual(output_1['code'], "test1") self.assert_(len(output_1['code']) <= 6) self.assertEqual(len(output_1['rules']), 4) self.assertEqual(output_1['rules'][0]['field'], '980.a') self.assertEqual(output_1['rules'][0]['template'], 'Picture_HTML_detailed.bft') self.assertEqual(output_1['rules'][0]['value'], 'PICTURE ') self.assertEqual(output_1['rules'][1]['field'], '980.a') self.assertEqual(output_1['rules'][1]['template'], 'Article.bft') self.assertEqual(output_1['rules'][1]['value'], 'ARTICLE') self.assertEqual(output_1['rules'][2]['field'], '980__a') self.assertEqual(output_1['rules'][2]['template'], 'Thesis_detailed.bft') self.assertEqual(output_1['rules'][2]['value'], 'THESIS ') self.assertEqual(output_1['rules'][3]['field'], '980__a') self.assertEqual(output_1['rules'][3]['template'], 'Pub.bft') self.assertEqual(output_1['rules'][3]['value'], 'PUBLICATION ') output_2 = ofs['test2'] #self.assertEqual(output_2['attrs']['names']['generic'], "") #self.assert_(isinstance(output_2['attrs']['names']['ln'], dict)) #self.assert_(isinstance(output_2['attrs']['names']['sn'], dict)) self.assertEqual(output_2['code'], "test2") self.assert_(len(output_2['code']) <= 6) self.assertEqual(output_2['rules'], []) try: unknown_output = bibformat_engine.get_output_format("unknow") except bibformat_engine.InvenioBibFormatError: pass else: self.fail("Should have raised the InvenioBibFormatError") def test_get_output_formats(self): """ bibformat - loading multiple output formats """ outputs = bibformat_engine.get_output_formats() self.assert_(isinstance(outputs, dict)) self.assert_("test1" in outputs.keys()) self.assert_("test2" in outputs.keys()) self.assert_("unknow" not in outputs.keys()) # Test correct parsing output_1 = outputs["test1"] #self.assertEqual(output_1['attrs']['names']['generic'], "") #self.assert_(isinstance(output_1['attrs']['names']['ln'], dict)) #self.assert_(isinstance(output_1['attrs']['names']['sn'], dict)) self.assertEqual(output_1['code'], "test1") self.assert_(len(output_1['code']) <= 6) class PatternTest(InvenioTestCase): """ bibformat - tests on re patterns""" def test_pattern_lang(self): """ bibformat - correctness of pattern 'pattern_lang'""" text = '''

    Here is my test text

    Some wordsQuelques motsEinige Wörter garbage Here ends the middle of my test text EnglishFrançaisDeutsch Here ends my test text

    ''' result = bibformat_engine.pattern_lang.search(text) self.assertEqual(result.group("langs"), "Some wordsQuelques motsEinige Wörter garbage ") text = '''

    Here is my test text

    ''' result = bibformat_engine.pattern_lang.search(text) self.assertEqual(result.group("langs"), "Some wordsQuelques motsEinige Wörter garbage ") def test_ln_pattern(self): """ bibformat - correctness of pattern 'ln_pattern'""" text = "Some wordsQuelques motsEinige Wörter garbage " result = bibformat_engine.ln_pattern.search(text) self.assertEqual(result.group(1), "en") self.assertEqual(result.group(2), "Some words") def test_pattern_format_template_name(self): """ bibformat - correctness of pattern 'pattern_format_template_name'""" text = ''' garbage a name a description on 2 lines

    the content of the template

    content ''' result = bibformat_engine.pattern_format_template_name.search(text) self.assertEqual(result.group('name'), "a name") def test_pattern_format_template_desc(self): """ bibformat - correctness of pattern 'pattern_format_template_desc'""" text = ''' garbage a name a description on 2 lines

    the content of the template

    content ''' result = bibformat_engine.pattern_format_template_desc.search(text) self.assertEqual(result.group('desc'), '''a description on 2 lines ''') def test_pattern_tag(self): """ bibformat - correctness of pattern 'pattern_tag'""" text = ''' garbage but part of content a name a description on 2 lines

    the content of the template

    my content is so nice! ''' result = bibformat_engine.pattern_tag.search(text) self.assertEqual(result.group('function_name'), "tiTLE") self.assertEqual(result.group('params').strip(), '''param1="value1" param2=""''') def test_pattern_function_params(self): """ bibformat - correctness of pattern 'test_pattern_function_params'""" text = ''' param1="" param2="value2" param3="value3" garbage ''' names = ["param1", "param2", "param3"] values = ["", "value2", "value3"] results = bibformat_engine.pattern_format_element_params.finditer(text) #TODO param_i = 0 for match in results: self.assertEqual(match.group('param'), names[param_i]) self.assertEqual(match.group('value'), values[param_i]) param_i += 1 def test_pattern_format_element_params(self): """ bibformat - correctness of pattern 'pattern_format_element_params'""" text = ''' a description for my element some text @param param1: desc1 @param param2: desc2 @see: seethis, seethat ''' names = ["param1", "param2"] descriptions = ["desc1", "desc2"] results = bibformat_engine.pattern_format_element_params.finditer(text) #TODO param_i = 0 for match in results: self.assertEqual(match.group('name'), names[param_i]) self.assertEqual(match.group('desc'), descriptions[param_i]) param_i += 1 def test_pattern_format_element_seealso(self): """ bibformat - correctness of pattern 'pattern_format_element_seealso' """ text = ''' a description for my element some text @param param1: desc1 @param param2: desc2 @see: seethis, seethat ''' result = bibformat_engine.pattern_format_element_seealso.search(text) self.assertEqual(result.group('see').strip(), 'seethis, seethat') class EscapingAndWashingTest(InvenioTestCase): """ bibformat - test escaping and washing metadata""" def test_escaping(self): """ bibformat - tests escaping HTML characters""" text = "Is 5 < 6 ? For sure! And what about True && False == True?" result = bibformat_engine.escape_field(text, mode=0) self.assertEqual(result, text) result = bibformat_engine.escape_field(text, mode=1) self.assertEqual(result, 'Is 5 < 6 ? For sure! And what about True && False == True?') def test_washing(self): """ bibformat - test washing HTML tags""" text = '''Hi dude,
    , please login:
    login here''' # Keep only basic tags result = bibformat_engine.escape_field(text, mode=2) self.assert_('script' not in result.lower()) self.assert_('onclick' not in result.lower()) self.assert_('mycrappywebsite' not in result.lower()) self.assert_('
    ' in result.lower()) self.assert_('
    ' in result.lower().replace(' ', '')) # Keep only basic tags only if value starts with # directive. Otherwise escape (which is the case here) result = bibformat_engine.escape_field(text, mode=3) self.assert_('' not in result.lower()) self.assert_('
    ' not in result.lower().replace(' ', '')) self.assert_('login here' in result.lower()) # Keep basic tags + some others (like ) result = bibformat_engine.escape_field(text, mode=5) self.assert_('script' not in result.lower()) self.assert_('onclick' not in result.lower()) self.assert_('mycrappywebsite' not in result.lower()) self.assert_('' result = bibformat_engine.escape_field(text2, mode=5) self.assert_(' # directive. Otherwise escape (which is the case here) result = bibformat_engine.escape_field(text, mode=6) self.assert_(''+text, mode=6) self.assert_('' in result.lower()) self.assert_('mycrappywebsite' not in result.lower()) # When the value cannot be parsed by our not so smart parser, # just escape everything text3 = """Ok, let't try with something unparsable < hehe login""" result = bibformat_engine.escape_field(text3, mode=2) self.assert_('mycrappywebsite' not in result.lower() or ' 33 thesis Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 99999 ''' #rec_1 = bibrecord.create_record(self.xml_text_1) self.bfo_1 = bibformat_engine.BibFormatObject(recID=None, ln='fr', xml_record=self.xml_text_1) self.xml_text_2 = ''' 33 thesis Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 ''' #self.rec_2 = bibrecord.create_record(xml_text_2) self.bfo_2 = bibformat_engine.BibFormatObject(recID=None, ln='fr', xml_record=self.xml_text_2) self.xml_text_3 = ''' 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 article ''' #self.rec_3 = bibrecord.create_record(xml_text_3) self.bfo_3 = bibformat_engine.BibFormatObject(recID=None, ln='fr', xml_record=self.xml_text_3) self.empty_record_xml = ''' 555 ''' self.no_001_record_xml = ''' eng Doe1, John ''' self.app.extensions['registry']['output_formats_directories'] = \ output_formats_directories_registry() from invenio.modules.formatter.registry import output_formats as ofs ofs.expunge() - self.app.extensions['registry']['format_elements'] = format_elements_registry() - self.app.extensions['registry']['format_templates'] = format_templates_registry() - from invenio.modules.formatter.registry import format_templates_lookup - format_templates_lookup.expunge() - #self.old_import_path = cfg['CFG_BIBFORMAT_ELEMENTS_IMPORT_PATH'] - #cfg['CFG_BIBFORMAT_ELEMENTS_IMPORT_PATH'] = CFG_BIBFORMAT_ELEMENTS_IMPORT_PATH - self.old_templates_path = cfg['CFG_BIBFORMAT_TEMPLATES_PATH'] - cfg['CFG_BIBFORMAT_TEMPLATES_PATH'] = format_templates.__path__[0] def tearDown(self): sys.path.pop() del self.app.extensions['registry']['output_formats_directories'] from invenio.modules.formatter.registry import output_formats output_formats.expunge() - from invenio.modules.formatter.registry import format_templates_lookup - format_templates_lookup.expunge() - del self.app.extensions['registry']['format_elements'] - #cfg['CFG_BIBFORMAT_ELEMENTS_IMPORT_PATH'] = self.old_import_path - cfg['CFG_BIBFORMAT_TEMPLATES_PATH'] = self.old_templates_path def test_decide_format_template(self): """ bibformat - choice made by function decide_format_template""" result = bibformat_engine.decide_format_template(self.bfo_1, "test1") self.assertEqual(result, "Thesis_detailed.bft") result = bibformat_engine.decide_format_template(self.bfo_3, "test3") - self.assertEqual(result, "Test3.bft") + self.assertEqual(result, "Test3.tpl") #Only default matches result = bibformat_engine.decide_format_template(self.bfo_2, "test1") self.assertEqual(result, "Default_HTML_detailed.bft") #No match at all for record result = bibformat_engine.decide_format_template(self.bfo_2, "test2") self.assertEqual(result, None) #Non existing output format try: result = bibformat_engine.decide_format_template(self.bfo_2, "UNKNOW") except bibformat_engine.InvenioBibFormatError: pass else: self.fail("Should have raised InvenioBibFormatError") def test_format_record(self): """ bibformat - correct formatting""" + from jinja2 import TemplateNotFound #use output format that has no match TEST DISABLED DURING MIGRATION #result = bibformat_engine.format_record(recID=None, of="test2", xml_record=self.xml_text_2) #self.assertEqual(result.replace("\n", ""),"") #use output format that link to unknown template - result, needs_2nd_pass = bibformat_engine.format_record(recID=None, of="test3", xml_record=self.xml_text_2) - self.assertEqual(result.replace("\n", ""), "") - self.assertEqual(needs_2nd_pass, False) + self.assertRaises(TemplateNotFound, bibformat_engine.format_record, + recID=None, of="test3", xml_record=self.xml_text_2) #Unknown output format TEST DISABLED DURING MIGRATION #result = bibformat_engine.format_record(recID=None, of="unkno", xml_record=self.xml_text_3) #self.assertEqual(result.replace("\n", ""),"") #Default formatting result, needs_2nd_pass = bibformat_engine.format_record(recID=None, ln='fr', of="test3", xml_record=self.xml_text_3) - self.assertEqual(result, '''

    hi

    this is my template\ntesttfrgarbage\n
    test me!<b>ok</b>a default valueeditor\n
    test me!oka default valueeditor\n
    test me!<b>ok</b>a default valueeditor\n''') + self.assertEqual(result, '''

    hi

    this is my template''') self.assertEqual(needs_2nd_pass, False) def test_empty_formatting(self): """bibformat - formatting empty record""" result = bibformat_engine.format_record(recID=0, of='hb', verbose=9, xml_record=self.empty_record_xml) self.assertEqual(result, ('', False)) # FIXME: The commented test below currently fails, since xm # format is generated from the database # result = bibformat_engine.format_record(recID=0, # of='xm', # verbose=9, # xml_record=self.empty_record_xml) # self.assertEqual(result, self.empty_record_xml) - def test_format_with_format_template(self): - """ bibformat - correct formatting with given template""" - del self.app.extensions['registry']['output_formats_directories'] - from invenio.modules.formatter.registry import output_formats - output_formats.expunge() - list(output_formats) - template = bibformat_engine.get_format_template("Test3.bft") - result, no_cache = bibformat_engine.format_with_format_template( - format_template_filename=None, - bfo=self.bfo_1, - verbose=0, - format_template_code=template['code']) - - self.assertEqual(result, '''

    hi

    this is my template\ntesttfrgarbage\n
    test me!<b>ok</b>a default valueeditor\n
    test me!oka default valueeditor\n
    test me!<b>ok</b>a default valueeditor\n99999''') - self.assertEqual(no_cache, False) - - - def test_format_2_passes_manually(self): - result, needs_2nd_pass = bibformat_engine.format_record( - recID=None, - of="test6", - xml_record=self.xml_text_2) - self.assertEqual(result, "\n") - self.assertEqual(needs_2nd_pass, True) - - out = bibformat_engine.format_record_2nd_pass(recID=None, - xml_record=self.xml_text_2, - template=result) - self.assertEqual(out, "helloworld\n") - - def test_format_translations_no_2nd_pass_en(self): - result, needs_2nd_pass = bibformat_engine.format_record( - recID=None, - of="test7", - xml_record=self.xml_text_2, - ln='en') - self.assertEqual(result.strip(), 'Title en\n') - self.assertEqual(needs_2nd_pass, False) - - def test_format_translations_no_2nd_pass_fr(self): - ln = 'fr' - result, needs_2nd_pass = bibformat_engine.format_record( - recID=None, - of="test7", - xml_record=self.xml_text_2, - ln=ln) - _ = gettext_set_language(ln) - self.assertEqual(result.strip(), 'Titre fr\n' % _('Record')) - self.assertEqual(needs_2nd_pass, False) - - def test_format_translations_with_2nd_pass_en(self): - result, needs_2nd_pass = bibformat_engine.format_record( - recID=None, - of="test8", - xml_record=self.xml_text_2, - ln='en') - self.assertEqual(result.strip(), '\n Title en\n Titre fr\n\n\n') - self.assertEqual(needs_2nd_pass, True) - - out = bibformat_engine.format_record_2nd_pass(recID=None, - template=result, - xml_record=self.xml_text_2, - ln='en') - self.assertEqual(out, 'Title en\nhelloworld\n') - - def test_format_translations_with_2nd_pass_fr(self): - ln = 'fr' - result, needs_2nd_pass = bibformat_engine.format_record( - recID=None, - of="test8", - xml_record=self.xml_text_2, - ln=ln) - _ = gettext_set_language(ln) - self.assertEqual(result.strip(), '\n Title en\n Titre fr\n\n\n') - self.assertEqual(needs_2nd_pass, True) - - out = bibformat_engine.format_record_2nd_pass(recID=None, - template=result, - xml_record=self.xml_text_2, - ln=ln) - self.assertEqual(out, 'Titre fr\nhelloworld\n' % _('Record')) - def test_engine_xslt_format(self): from ..engines import xslt template = pkg_resources.resource_filename( 'invenio.modules.formatter', 'format_templates/RSS.xsl') output = xslt.format(self.xml_text_1, template_filename=template) assert output.startswith( '\n On the foo and bar1On the foo and bar2\n' ' \n \n ' '' 'Doe2, John\n \n\n' ) def test_format_record_no_recid(self): from invenio.modules.formatter import format_record result = format_record(recID=None, of="test6", xml_record=self.no_001_record_xml) self.assertEqual(result, "helloworld\n") class MarcFilteringTest(InvenioTestCase): """ bibformat - MARC tag filtering tests""" def setUp(self): """bibformat - prepare MARC filtering tests""" self.xml_text_4 = ''' 33 eng Doe1, John Doe2, John editor On the foo and bar1 On the foo and bar2 Confidential comment article ''' def test_filtering(self): """bibformat - filter hidden fields""" newxml = bibformat_engine.filter_hidden_fields(self.xml_text_4, user_info=None, filter_tags=['595'], force_filtering=True) numhfields = newxml.count("595") self.assertEqual(numhfields, 0) newxml = bibformat_engine.filter_hidden_fields(self.xml_text_4, user_info=None, filter_tags=['595'], force_filtering=False) numhfields = newxml.count("595") self.assertEqual(numhfields, 1) class BibFormat2ndPassTest(InvenioTestCase): """Check for 2 passes parsing for record""" + @property + def config(self): + data = super(BibFormat2ndPassTest, self).config + data['PACKAGES'] = TEST_PACKAGES + data.get('PACKAGES', PACKAGES) + return data + def setUp(self): - self.app.extensions['registry']['format_templates'] = format_templates_registry() - self.app.extensions['registry']['format_elements'] = format_elements_registry() - self.app.extensions['registry']['output_formats_directories'] = output_formats_directories_registry() - from invenio.modules.formatter.registry import output_formats - output_formats.expunge() self.xml_text = ''' 33 thesis ''' - def tearDown(self): - from invenio.modules.formatter.registry import output_formats - output_formats.expunge() - del self.app.extensions['registry']['output_formats_directories'] - del self.app.extensions['registry']['format_templates'] - del self.app.extensions['registry']['format_elements'] - def test_format_2_passes(self): from invenio.modules.formatter import format_record result = format_record(recID=None, of="test6", xml_record=self.xml_text) self.assertEqual(result, "helloworld\n") -TEST_SUITE = make_test_suite(FormatTemplateTest, - OutputFormatTest, - FormatElementTest, +TEST_SUITE = make_test_suite(OutputFormatTest, PatternTest, MiscTest, FormatTest, EscapingAndWashingTest, MarcFilteringTest, BibFormat2ndPassTest) if __name__ == '__main__': run_test_suite(TEST_SUITE) diff --git a/invenio/modules/formatter/format_elements/bfe_language.py b/invenio/modules/formatter/testsuite/views.py similarity index 74% rename from invenio/modules/formatter/format_elements/bfe_language.py rename to invenio/modules/formatter/testsuite/views.py index 6f4d146cd..a73da4091 100644 --- a/invenio/modules/formatter/format_elements/bfe_language.py +++ b/invenio/modules/formatter/testsuite/views.py @@ -1,30 +1,27 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. -# Copyright (C) 2007, 2008, 2010, 2011 CERN. +# Copyright (C) 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. -"""BibFormat element - Prints the language of the document -""" -__revision__ = "$Id$" +"""Test Formater Blueprint.""" +from __future__ import unicode_literals -def format_element(bfo): - """ - Prints the language of the document. - """ +from flask import Blueprint - return bfo.field('041__a') +blueprint = Blueprint('test_formatter', __name__, + template_folder='templates', static_folder='static') diff --git a/invenio/modules/knowledge/api.py b/invenio/modules/knowledge/api.py index 9bf5960e0..89912e50a 100644 --- a/invenio/modules/knowledge/api.py +++ b/invenio/modules/knowledge/api.py @@ -1,817 +1,757 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2009, 2010, 2011, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Provide API-callable functions for knowledge base management.""" import json import os import re import warnings from invenio.base.globals import cfg from invenio.base.i18n import _ from invenio.ext.sqlalchemy import db from invenio.ext.sqlalchemy.utils import session_manager from invenio.modules.collections.models import Collection from invenio.utils.memoise import Memoise from sqlalchemy.exc import IntegrityError from sqlalchemy.orm.exc import NoResultFound from . import models processor_type = 0 try: from lxml import etree processor_type = 1 except ImportError: try: import libxml2 import libxslt processor_type = 2 except ImportError: pass def get_kb_by_slug(slug): """Return the knwKB object with given slug. :param slug: slug of knowledge :return: knowledge's object :raises: :exc:`~sqlalchemy.orm.exc.NoResultFound` in case not exist. """ return models.KnwKB.query.filter_by(slug=slug).one() def get_kb_by_id(kb_id): """Return the knwKB object with given id. :param kb_id: id of knowledge :return: knowledge's object :raises: :exc:`~sqlalchemy.orm.exc.NoResultFound` in case not exist. """ return models.KnwKB.query.filter_by(id=kb_id).one() def get_kb_id(kb_name): """Get the id by name. :param kb_name: knowledge base name """ warnings.warn("The method get_kb_id(kb_name) is deprecated! " "Use instead get_kb_by_id()'", DeprecationWarning) return get_kb_by_id(kb_name).id def get_kb_by_name(kb_name): """Return the knwKB object with given name. :raises: :exc:`~sqlalchemy.orm.exc.NoResultFound` in case not exist. """ return models.KnwKB.query.filter_by(name=kb_name).one() def get_all_kb_names(): """Return all knowledge base names. :return: list of names """ return [row.name for row in models.KnwKB.query.all()] get_kb_by_name_memoised = Memoise(get_kb_by_name) def query_get_kb_by_type(kbtype): """Return a query to filter kb by type. :param kbtype: type to filter (e.g: taxonomy) :return: query to filter kb """ return models.KnwKB.query.filter_by( kbtype=models.KnwKB.KNWKB_TYPES[kbtype]) def query_kb_mappings(kbid, sortby="to", key="", value="", match_type="s"): """Return a list of all mappings from the given kb, ordered by key. If key given, give only those with left side (mapFrom) = key. If value given, give only those with right side (mapTo) = value. :param kb_name: knowledge base name. if "", return all :param sortby: the sorting criteria ('from' or 'to') :param key: return only entries where key matches this :param value: return only entries where value matches this :param match_type: s=substring, e=exact, sw=startswith """ return models.KnwKBRVAL.query_kb_mappings(kbid, sortby, key, value, match_type) def get_kb_mappings(kb_name="", key="", value="", match_type="s", sortby="to", limit=None): """Return a list of all mappings from the given kb, ordered by key. If key given, give only those with left side (mapFrom) = key. If value given, give only those with right side (mapTo) = value. :param kb_name: knowledge base name. if "", return all :param sortby: the sorting criteria ('from' or 'to') :param key: return only entries where key matches this :param value: return only entries where value matches this :param limit: return only X number of entries :return: list of knowledge converted in dictionary """ # query query = db.session.query(models.KnwKBRVAL).join(models.KnwKB) # filter if kb_name: query = query.filter(models.KnwKB.name == kb_name) if len(key) > 0: if match_type == "s": key = "%"+key+"%" else: key = '%' if len(value) > 0: if match_type == "s": value = "%"+value+"%" else: value = '%' query = query.filter( models.KnwKBRVAL.m_key.like(key), models.KnwKBRVAL.m_value.like(value)) # order by if sortby == "from": query = query.order_by(models.KnwKBRVAL.m_key) else: query = query.order_by(models.KnwKBRVAL.m_value) if limit: query = query.limit(limit) # return results return [kbv.to_dict() for (kbv) in query.all()] def get_kb_mapping(kb_name="", key="", value="", match_type="e", default="", limit=None): """Get one unique mapping. If not found, return default. :param kb_name: the name of the kb :param key: include only lines matching this on left side in the results :param value: include only lines matching this on right side in the results :param match_type: s = substring match, e = exact match :param default: default value if no mapping is found :return: a mapping """ mappings = get_kb_mappings(kb_name, key=key, value=value, match_type=match_type, limit=limit) if len(mappings) == 0: return default else: return mappings[0] @session_manager def add_kb_mapping(kb_name, key, value=""): """Add a new mapping to given kb. :param kb_name: the name of the kb where to insert the new value :param key: the key of the mapping :param value: the value of the mapping """ kb = get_kb_by_name(kb_name) if key in kb.kbrvals: # update kb.kbrvals[key].m_value = value else: # insert kb.kbrvals.set(models.KnwKBRVAL(m_key=key, m_value=value)) @session_manager def remove_kb_mapping(kb_name, key): """Delete an existing kb mapping in kb. :param kb_name: the name of the kb where to insert the new value :param key: the key of the mapping """ kb = get_kb_by_name(kb_name) del kb.kbrvals[key] @session_manager def update_kb_mapping(kb_name, old_key, key, value): """Update an existing kb mapping with key old_key with a new key and value. :param kb_name: the name of the kb where to insert the new value :param old_key: the key of the mapping in the kb :param key: the new key of the mapping :param value: the new value of the mapping """ db.session.query(models.KnwKBRVAL).join(models.KnwKB) \ .filter(models.KnwKB.name == kb_name, models.KnwKBRVAL.m_key == old_key) \ .update({"m_key": key, "m_value": value}, synchronize_session=False) def get_kb_mappings_json(kb_name="", key="", value="", match_type="s", limit=None): """Get leftside/rightside mappings from kb kb_name formatted as json dict. If key given, give only those with left side (mapFrom) = key. If value given, give only those with right side (mapTo) = value. :param kb_name: the name of the kb :param key: include only lines matching this on left side in the results :param value: include only lines matching this on right side in the results :param match_type: s = substring match, e = exact match :param limit: maximum number of results to return (are ALL if set to None) :return: a list of mappings """ mappings = get_kb_mappings(kb_name, key, value, match_type) ret = [] if limit is None: limit = len(mappings) for m in mappings[:limit]: label = m['value'] or m['key'] value = m['key'] or m['value'] ret.append({'label': label, 'value': value}) return json.dumps(ret) def get_kb_mappings_embedded_json(kb_name="", key="", value="", match_type="s", limit=None): """Get leftside/rightside mappings from kb kb_name formatted as json dict. The rightside is actually considered as a json string and hence embedded within the final result. If key given, give only those with left side (mapFrom) = key. If value given, give only those with right side (mapTo) = value. :param kb_name: the name of the kb :param key: include only lines matching this on left side in the results :param value: include only lines matching this on right side in the results :param match_type: s = substring match, e = exact match :param limit: maximum number of results to return (are ALL if set to None) :return: a list of mappings """ mappings = get_kb_mappings(kb_name, key, value, match_type) ret = [] if limit is None: limit = len(mappings) for m in mappings[:limit]: label = m['value'] or m['key'] value = m['key'] or m['value'] ret.append({'label': label, 'value': json.loads(value)}) return json.dumps(ret) def kb_exists(kb_name): """Return True if a kb with the given name exists. :param kb_name: the name of the knowledge base :return: True if kb exists """ return models.KnwKB.exists(kb_name) def get_kb_name(kb_id): """Return the name of the kb given by id. :param kb_id: the id of the knowledge base """ return get_kb_by_id(kb_id).name @session_manager def update_kb_attributes(kb_name, new_name, new_description=''): """Update kb kb_name with a new name and (optionally) description. :param kb_name: the name of the kb to update :param new_name: the new name for the kb :param new_description: the new description for the kb """ models.KnwKB.query.filter_by(name=kb_name) \ .update({"name": new_name, "description": new_description}) def add_kb(kb_name=u"Untitled", kb_type=None, tries=10): """Add a new kb in database, return the id. Add a new kb in database, and returns its id The name of the kb will be 'Untitled#' such that it is unique. :param kb_name: the name of the kb :param kb_type: the type of the kb, incl 'taxonomy' and 'dynamic'. None for typical (leftside-rightside). :param tries: exit after retry :return: the id of the newly created kb """ created = False name = kb_name i = 0 while(i < tries and created is False): try: kb = models.KnwKB(name=name, description="", kbtype=kb_type) created = True db.session.add(kb) db.session.commit() except IntegrityError: db.session.rollback() # get the highest id to calculate the new name result = db.session.execute( db.select([models.KnwKB.id]) .order_by(db.desc(models.KnwKB.id)) .limit(1)).first() index = result[0] + 1 if result is not None else 1 name = kb_name + " " + str(index) i = i + 1 created = False except Exception: db.session.rollback() raise if created is False: # TODO raise the right exception raise Exception(_("Can't create knowledge base \"%(name)s\".\n" "Probabily the server is busy! " "Try again later.", name=kb_name)) return kb.id def add_dynamic_kb(kbname, tag, collection="", searchwith=""): """A convenience method.""" kb_id = add_kb(kb_name=kbname, kb_type='dynamic') save_kb_dyn_config(kb_id, tag, searchwith, collection) return kb_id def save_kb_dyn_config(kb_id, field, expression, collection=None): """Save a dynamic knowledge base configuration. :param kb_id: the id :param field: the field where values are extracted :param expression: ..using this expression :param collection: ..in a certain collection (default is all) """ # check that collection exists if collection: collection = Collection.query.filter_by(name=collection).one() kb = get_kb_by_id(kb_id) kb.set_dyn_config(field, expression, collection) def kb_mapping_exists(kb_name, key): """Return the information if a mapping exists. :param kb_name: knowledge base name :param key: left side (mapFrom) """ try: kb = get_kb_by_name(kb_name) except NoResultFound: return False return key in kb.kbrvals @session_manager def delete_kb(kb_name): """Delete given kb from database. :param kb_name: knowledge base name """ db.session.delete(models.KnwKB.query.filter_by( name=kb_name).one()) - -# Knowledge Bases Dependencies -# - - -def get_elements_that_use_kb(name): - """Return a list of elements that call given kb. - - WARNING: this routine is obsolete. - - .. code-block:: python - - [ - { - 'filename':"filename_1.py", - 'name': "a name" - }, - .. - ] - - :return: elements sorted by name - """ - # FIXME remove the obsolete function - warnings.warn("The method 'get_elements_that_use_kb(name) is obsolete!'", - DeprecationWarning) - - format_elements = {} - # Retrieve all elements in files - from invenio.modules.formatter.engine \ - import TEMPLATE_CONTEXT_FUNCTIONS_CACHE - for element in TEMPLATE_CONTEXT_FUNCTIONS_CACHE \ - .bibformat_elements().values(): - path = element.__file__ - filename = os.path.basename(element.__file__) - if filename.endswith(".py"): - formatf = open(path, 'r') - code = formatf.read() - formatf.close() - # Search for use of kb inside code - kb_pattern = re.compile(''' - (bfo.kb)\s* #Function call - \(\s* #Opening parenthesis - [\'"]+ #Single or double quote - (?P%s) #kb - [\'"]+\s* #Single or double quote - , #comma - ''' % name, re.VERBOSE | re.MULTILINE | re.IGNORECASE) - - result = kb_pattern.search(code) - if result is not None: - name = ("".join(filename.split(".")[:-1])).lower() - if name.startswith("bfe_"): - name = name[4:] - format_elements[name] = {'filename': filename, - 'name': name} - - keys = format_elements.keys() - keys.sort() - return map(format_elements.get, keys) - # kb functions for export def get_kbs_info(kbtype="", searchkbname=""): """A convenience method. :param kbtype: type of kb -- get only kb's of this type :param searchkbname: get only kb's where this sting appears in the name """ # query + order by query = models.KnwKB.query.order_by( models.KnwKB.name) # filters if kbtype: query = query.filter_by(kbtype=kbtype) if searchkbname: query = query.filter_by(name=searchkbname) return [row.to_dict() for row in query.all()] def get_kba_values(kb_name, searchname="", searchtype="s"): """Return an array of values "authority file" type = just values. :param kb_name: name of kb :param searchname: get these values, according to searchtype :param searchtype: s=substring, e=exact, , sw=startswith """ if searchtype == 's' and searchname: searchname = '%'+searchname+'%' if searchtype == 'sw' and searchname: # startswith searchname = searchname+'%' if not searchname: searchname = '%' query = db.session.query(models.KnwKBRVAL).join(models.KnwKB) \ .filter(models.KnwKBRVAL.m_value.like(searchname), models.KnwKB.name.like(kb_name)) return [(k.m_value,) for k in query.all()] def get_kbr_keys(kb_name, searchkey="", searchvalue="", searchtype='s'): """Return an array of keys. :param kb_name: the name of the knowledge base :param searchkey: search using this key :param searchvalue: search using this value :param searchtype: s = substring, e=exact """ if searchtype == 's' and searchkey: searchkey = '%'+searchkey+'%' if searchtype == 's' and searchvalue: searchvalue = '%'+searchvalue+'%' if searchtype == 'sw' and searchvalue: # startswith searchvalue = searchvalue+'%' if not searchvalue: searchvalue = '%' if not searchkey: searchkey = '%' query = db.session.query(models.KnwKBRVAL).join(models.KnwKB) \ .filter(models.KnwKBRVAL.m_key.like(searchkey), models.KnwKBRVAL.m_value.like(searchvalue), models.KnwKB.name.like(kb_name)) return [(k.m_key,) for k in query.all()] def get_kbr_values(kb_name, searchkey="", searchvalue="", searchtype='s', use_memoise=False): """Return a tuple of values from key-value mapping kb. :param kb_name: the name of the knowledge base :param searchkey: search using this key :param searchvalue: search using this value :param searchtype: s=substring; e=exact :param use_memoise: can we memoise while doing lookups? :type use_memoise: bool """ try: if use_memoise: kb = get_kb_by_name_memoised(kb_name) else: kb = get_kb_by_name(kb_name) except NoResultFound: return [] return list(kb.get_kbr_values(searchkey, searchvalue, searchtype)) def get_kbr_items(kb_name, searchkey="", searchvalue="", searchtype='s'): """Return a list of dictionaries that match the search. :param kb_name: the name of the knowledge base :param searchkey: search using this key :param searchvalue: search using this value :param searchtype: s = substring, e=exact :return: a list of dictionaries [{'key'=>x, 'value'=>y},..] """ kb = get_kb_by_name(kb_name) return kb.get_kbr_items(searchkey, searchvalue, searchtype) def get_kbd_values(kbname, searchwith=""): """Return a list of values by searching a dynamic kb. :param kbname: name of the knowledge base :param searchwith: a term to search with :return: list of values """ kb = get_kb_by_name(kbname) kbid = kb.id if not kbid: return [] kbtype = kb.kbtype if not kbtype: return [] if kbtype != 'd': return [] return get_kbd_values_by_def(kb.kbdefs.to_dict(), searchwith) def get_kbd_values_by_def(confdict, searchwith=""): """Return a list of values by searching a dynamic kb. :param confdict: dictionary with keys "field", "expression" and "collection" name :param searchwith: a term to search with :return: list of values """ from invenio.legacy import search_engine # get the configuration so that we see what the field is if not confdict: return [] if 'field' not in confdict: return [] field = confdict['field'] expression = confdict['expression'] collection = "" if 'collection' in confdict: collection = confdict['collection'] reclist = [] # return this if searchwith and expression: if (expression.count('%') > 0): expression = expression.replace("%", searchwith) reclist = search_engine.perform_request_search(p=expression, cc=collection) else: # no %.. just make a combination expression = expression + " and " + searchwith reclist = search_engine.perform_request_search(p=expression, cc=collection) else: # either no expr or no searchwith.. but never mind about searchwith if expression: # in this case: only expression reclist = search_engine.perform_request_search(p=expression, cc=collection) else: # make a fake expression so that only records that have this field # will be returned fake_exp = "/.*/" if searchwith: fake_exp = searchwith reclist = search_engine.perform_request_search(f=field, p=fake_exp, cc=collection) if reclist: return [val for (val, dummy) in search_engine.get_most_popular_field_values(reclist, field)] return [] # in case nothing worked def get_kbd_values_json(kbname, searchwith=""): """Return values from searching a dynamic kb as a json-formatted string. This IS probably the method you want. :param kbname: name of the knowledge base :param searchwith: a term to search with """ res = get_kbd_values(kbname, searchwith) return json.dumps(res) def get_kbd_values_for_bibedit(tag, collection="", searchwith="", expression=""): """Get list of kbd values for bibedit. Example1: tag=100__a : return values of 100__a Example2: tag=100__a, searchwith=Jill: return values of 100__a that match with Jill Example3: tag=100__a, searchwith=Ellis, expression="700__a:*%*: return values of 100__a for which Ellis matches some 700__a Note: the performace of this function is ok compared to a plain perform_request_search / get most popular fields -pair. The overhead is about 5% with large record sets; the lookups are the xpensive part. :param tag: the tag like 100__a :param collection: collection id :param searchwith: the string to search. If empty, match all. :param expression: the search expression for perform_request_search; if present, '%' is substituted with /searcwith/. If absent, /searchwith/ is searched for in /tag/. """ return get_kbd_values_by_def(dict( field=tag, expression=expression, collection=collection ), searchwith) def get_kbt_items(taxonomyfilename, templatefilename, searchwith=""): """ Get items from taxonomy file using a templatefile. If searchwith is defined, return only items that match with it. :param taxonomyfilename: full path+name of the RDF file :param templatefile: full path+name of the XSLT file :param searchwith: a term to search with """ if processor_type == 1: # lxml doc = etree.XML(taxonomyfilename) styledoc = etree.XML(templatefilename) style = etree.XSLT(styledoc) result = style(doc) strres = str(result) del result del style del styledoc del doc elif processor_type == 2: # libxml2 & libxslt styledoc = libxml2.parseFile(templatefilename) style = libxslt.parseStylesheetDoc(styledoc) doc = libxml2.parseFile(taxonomyfilename) result = style.applyStylesheet(doc, None) strres = style.saveResultToString(result) style.freeStylesheet() doc.freeDoc() result.freeDoc() else: # no xml parser found strres = "" ritems = [] if len(strres) == 0: return [] else: lines = strres.split("\n") for line in lines: if searchwith: if line.count(searchwith) > 0: ritems.append(line) else: if len(line) > 0: ritems.append(line) return ritems def get_kbt_items_for_bibedit(kbtname, tag="", searchwith=""): """A simplifield, customized version of the function get_kbt_items. Traverses an RDF document. By default returns all leaves. If tag defined returns the content of that tag. If searchwith defined, returns leaves that match it. Warning! In order to make this faster, the matching field values cannot be multi-line! :param kbtname: name of the taxonony kb :param tag: name of tag whose content :param searchwith: a term to search with """ # get the actual file based on the kbt name kb = get_kb_by_name(kbtname) kb_id = kb.id if not kb_id: return [] # get the rdf file.. rdfname = cfg['CFG_WEBDIR'] + "/kbfiles/" + str(kb_id) + ".rdf" if not os.path.exists(rdfname): return [] xsl = """\ """ if processor_type == 1: styledoc = etree.XML(xsl) style = etree.XSLT(styledoc) doc = etree.parse(open(rdfname, 'r')) strres = str(style(doc)) elif processor_type == 2: styledoc = libxml2.parseDoc(xsl) style = libxslt.parseStylesheetDoc(styledoc) doc = libxml2.parseFile(rdfname) result = style.applyStylesheet(doc, None) strres = style.saveResultToString(result) style.freeStylesheet() doc.freeDoc() result.freeDoc() else: # no xml parser found strres = "" ritems = [] if len(strres) == 0: return [] else: lines = strres.split("\n") for line in lines: # take only those with myout.. if line.count("") > 0: # remove the myout tag.. line = line[9:] line = line[:-8] if searchwith: if line.count(searchwith) > 0: ritems.append(line) else: ritems.append(line) return ritems if __name__ == "__main__": pass diff --git a/invenio/modules/workflows/models.py b/invenio/modules/workflows/models.py index d439f90ad..1a975d973 100644 --- a/invenio/modules/workflows/models.py +++ b/invenio/modules/workflows/models.py @@ -1,964 +1,956 @@ # -*- coding: utf-8 -*- # This file is part of Invenio. # Copyright (C) 2012, 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Models for BibWorkflow Objects.""" import base64 import logging import os import tempfile from datetime import datetime from invenio.base.globals import cfg from invenio.base.helpers import unicodifier -from invenio.base.utils import classproperty -from invenio.utils.deprecation import deprecated, RemovedInInvenio22Warning from invenio.ext.sqlalchemy import db from invenio.ext.sqlalchemy.utils import session_manager from six import callable, iteritems from six.moves import cPickle from sqlalchemy import desc from sqlalchemy.orm.exc import NoResultFound from .logger import BibWorkflowLogHandler, get_logger class ObjectVersion(object): """Specify the different versions possible.""" INITIAL = 0 COMPLETED = 1 HALTED = 2 RUNNING = 3 WAITING = 4 ERROR = 5 MAPPING = {"New": 0, "Done": 1, "Need action": 2, "In process": 3, "Waiting": 4, "Error": 5} - @classproperty - @deprecated("Please use ObjectVersion.COMPLETED " - "instead of ObjectVersion.FINAL", RemovedInInvenio22Warning) - def FINAL(cls): - return cls.COMPLETED - @classmethod def name_from_version(cls, version): try: return cls.MAPPING.keys()[cls.MAPPING.values().index(version)] except ValueError: return None def get_default_data(): """Return the base64 representation of the data default value.""" data_default = {} return base64.b64encode(cPickle.dumps(data_default)) def get_default_extra_data(): """Return the base64 representation of the extra_data default value.""" extra_data_default = {"_tasks_results": {}, "owner": {}, "_task_counter": {}, "_error_msg": None, "_last_task_name": "", "latest_object": -1, "_action": None, "redis_search": {}, "source": "", "_task_history": []} return base64.b64encode(cPickle.dumps(extra_data_default)) class Workflow(db.Model): """Represents a workflow instance. Used by BibWorkflowEngine to store the state of the workflow. """ __tablename__ = "bwlWORKFLOW" uuid = db.Column(db.String(36), primary_key=True, nullable=False) name = db.Column(db.String(255), default="Default workflow", nullable=False) created = db.Column(db.DateTime, default=datetime.now, nullable=False) modified = db.Column(db.DateTime, default=datetime.now, onupdate=datetime.now, nullable=False) id_user = db.Column(db.Integer, default=0, nullable=False) _extra_data = db.Column(db.LargeBinary, nullable=False, default=get_default_extra_data()) status = db.Column(db.Integer, default=0, nullable=False) current_object = db.Column(db.Integer, default="0", nullable=False) objects = db.relationship("BibWorkflowObject", backref='bwlWORKFLOW', cascade="all, delete, delete-orphan") counter_initial = db.Column(db.Integer, default=0, nullable=False) counter_halted = db.Column(db.Integer, default=0, nullable=False) counter_error = db.Column(db.Integer, default=0, nullable=False) counter_finished = db.Column(db.Integer, default=0, nullable=False) module_name = db.Column(db.String(64), nullable=False) child_logs = db.relationship("BibWorkflowEngineLog", backref='bwlWORKFLOW', cascade="all, delete, delete-orphan") def __repr__(self): """Represent a workflow object.""" return "" % \ (str(self.name), str(self.module_name), str(self.created), str(self.modified), str(self.id_user), str(self.status)) def __str__(self): """Print a workflow object.""" return """Workflow: Uuid: %s Name: %s User id: %s Module name: %s Created: %s Modified: %s Status: %s Current object: %s Counters: initial=%s, halted=%s, error=%s, finished=%s Extra data: %s""" % (str(self.uuid), str(self.name), str(self.id_user), str(self.module_name), str(self.created), str(self.modified), str(self.status), str(self.current_object), str(self.counter_initial), str(self.counter_halted), str(self.counter_error), str(self.counter_finished), str(self._extra_data)) @classmethod def get(cls, *criteria, **filters): """Wrapper to get a specified object. A wrapper for the filter and filter_by functions of sqlalchemy. Define a dict with which columns should be filtered by which values. .. code-block:: python Workflow.get(uuid=uuid) Workflow.get(Workflow.uuid != uuid) The function supports also "hybrid" arguments. .. code-block:: python Workflow.get(Workflow.module_name != 'i_hate_this_module', user_id=user_id) See also SQLAlchemy BaseQuery's filter and filter_by documentation. """ return cls.query.filter(*criteria).filter_by(**filters) @classmethod def get_status(cls, uuid=None): """Return the status of the workflow.""" return cls.get(Workflow.uuid == uuid).one().status @classmethod def get_most_recent(cls, *criteria, **filters): """Return the most recently modified workflow.""" most_recent = cls.get(*criteria, **filters). \ order_by(desc(Workflow.modified)).first() if most_recent is None: raise NoResultFound else: return most_recent @classmethod def get_objects(cls, uuid=None): """Return the objects of the workflow.""" return cls.get(Workflow.uuid == uuid).one().objects def get_extra_data(self, user_id=0, uuid=None, key=None, getter=None): """Get the extra_data for the object. Returns a JSON of the column extra_data or if any of the other arguments are defined, a specific value. You can define either the key or the getter function. :param key: the key to access the desirable value :param getter: a callable that takes a dict as param and returns a value """ extra_data = Workflow.get(Workflow.id_user == self.id_user, Workflow.uuid == self.uuid).one()._extra_data extra_data = cPickle.loads(base64.b64decode(extra_data)) if key: return extra_data[key] elif callable(getter): return getter(extra_data) elif not key: return extra_data def set_extra_data(self, user_id=0, uuid=None, key=None, value=None, setter=None): """Replace extra_data. Modifies the JSON of the column extra_data or if any of the other arguments are defined, a specific value. You can define either the key, value or the setter function. :param key: the key to access the desirable value :param value: the new value :param setter: a callable that takes a dict as param and modifies it """ extra_data = Workflow.get(Workflow.id_user == user_id, Workflow.uuid == uuid).one()._extra_data extra_data = cPickle.loads(base64.b64decode(extra_data)) if key is not None and value is not None: extra_data[key] = value elif callable(setter): setter(extra_data) Workflow.get(Workflow.uuid == self.uuid).update( {'_extra_data': base64.b64encode(cPickle.dumps(extra_data))} ) @classmethod @session_manager def delete(cls, uuid=None): """Delete a workflow.""" cls.get(Workflow.uuid == uuid).delete() @session_manager def save(self, status): """Save object to persistent storage.""" self.modified = datetime.now() if status is not None: self.status = status db.session.add(self) class BibWorkflowObject(db.Model): """Data model for wrapping data being run in the workflows. Main object being passed around in the workflows module when using the workflows API. It can be instantiated like this: .. code-block:: python obj = BibWorkflowObject() obj.save() Or, like this: .. code-block:: python obj = BibWorkflowObject.create_object() BibWorkflowObject provides some handy functions such as: .. code-block:: python obj.set_data("") obj.get_data() == "" obj.set_extra_data({"param": value}) obj.get_extra_data() == {"param": value} obj.add_task_result("myresult", {"result": 1}) Then to finally save the object .. code-block:: python obj.save() Now you can for example run it in a workflow: .. code-block:: python obj.start_workflow("sample_workflow") """ # db table definition __tablename__ = "bwlOBJECT" id = db.Column(db.Integer, primary_key=True) # Our internal data column. Default is encoded dict. _data = db.Column(db.LargeBinary, nullable=False, default=get_default_data()) _extra_data = db.Column(db.LargeBinary, nullable=False, default=get_default_extra_data()) id_workflow = db.Column(db.String(36), db.ForeignKey("bwlWORKFLOW.uuid"), nullable=True) version = db.Column(db.Integer(3), default=ObjectVersion.INITIAL, nullable=False) id_parent = db.Column(db.Integer, db.ForeignKey("bwlOBJECT.id"), default=None) child_objects = db.relationship("BibWorkflowObject", remote_side=[id_parent]) created = db.Column(db.DateTime, default=datetime.now, nullable=False) modified = db.Column(db.DateTime, default=datetime.now, onupdate=datetime.now, nullable=False) status = db.Column(db.String(255), default="", nullable=False) data_type = db.Column(db.String(150), default="", nullable=True) uri = db.Column(db.String(500), default="") id_user = db.Column(db.Integer, default=0, nullable=False) child_logs = db.relationship("BibWorkflowObjectLog", backref='bibworkflowobject', cascade="all, delete, delete-orphan") workflow = db.relationship( Workflow, foreign_keys=[id_workflow], remote_side=Workflow.uuid, ) _log = None @property def log(self): """Access logger object for this instance.""" if not self._log: db_handler_obj = BibWorkflowLogHandler(BibWorkflowObjectLog, "id") self._log = get_logger(logger_name="object.%s" % (self.id,), db_handler_obj=db_handler_obj, loglevel=logging.DEBUG, obj=self) return self._log def get_data(self): """Get data saved in the object.""" return cPickle.loads(base64.b64decode(self._data)) def set_data(self, value): """Save data to the object.""" self._data = base64.b64encode(cPickle.dumps(value)) def get_extra_data(self): """Get extra data saved to the object.""" return cPickle.loads(base64.b64decode(self._extra_data)) def set_extra_data(self, value): """Save extra data to the object. :param value: what you want to replace extra_data with. :type value: dict """ self._extra_data = base64.b64encode(cPickle.dumps(value)) def get_workflow_name(self): """Return the workflow name for this object.""" try: if self.id_workflow: return Workflow.query.get(self.id_workflow).name except AttributeError: # Workflow non-existent pass return def get_formatted_data(self, of="hd"): """Get the formatted representation for this object.""" from .registry import workflows try: name = self.get_workflow_name() if not name: return "" workflow_definition = workflows[name] formatted_data = workflow_definition.formatter( self, of=of ) except (KeyError, AttributeError): # Somehow the workflow or formatter does not exist from invenio.ext.logging import register_exception register_exception(alert_admin=True) formatted_data = "" return formatted_data def __repr__(self): """Represent a BibWorkflowObject.""" return "", if not specified. """ if text is None: text = "Filename: %s" % filename # Generate simple PDF from reportlab.pdfgen import canvas output = StringIO.StringIO() c = canvas.Canvas(output) c.drawString(100, 100, text) c.showPage() c.save() return make_file_fixture(filename, stringio_to_base64(output)) class InvenioTestUtilsBrowserException(Exception): """Exception for the regression test suite browser.""" pass # @nottest def test_web_page_existence(url): """ Test whether URL exists and is well accessible. :return: True or raise exception in case of problems. """ import mechanize browser = mechanize.Browser() try: browser.open(url) except Exception: raise return True def get_authenticated_mechanize_browser(username="guest", password=""): """Return an instance of an authenticated mechanize browser.""" try: import mechanize except ImportError: raise InvenioTestUtilsBrowserException( 'ERROR: Cannot import mechanize.') browser = mechanize.Browser() browser.set_handle_robots(False) # ignore robots.txt, since we test gently if username == "guest": return browser from invenio.config import CFG_SITE_SECURE_URL browser.open(CFG_SITE_SECURE_URL + "/youraccount/login") browser.select_form(nr=0) browser['nickname'] = username browser['password'] = password browser.submit() username_account_page_body = browser.response().read() try: username_account_page_body.index("You are logged in as %s." % username) except ValueError: raise InvenioTestUtilsBrowserException( 'ERROR: Cannot login as %s.' % username) return browser # @nottest def test_web_page_content(url, username="guest", password="", expected_text="", unexpected_text="", expected_link_target=None, expected_link_label=None, require_validate_p=None): """Test the content of a web page. Test whether web page URL as seen by user USERNAME contains text EXPECTED_TEXT and, eventually, contains a link to EXPECTED_LINK_TARGET (if set) labelled EXPECTED_LINK_LABEL (if set). The EXPECTED_TEXT is checked via substring matching, the EXPECTED_LINK_TARGET and EXPECTED_LINK_LABEL via exact string matching. EXPECTED_TEXT, EXPECTED_LINK_LABEL and EXPECTED_LINK_TARGET can either be strings or list of strings (in order to check multiple values inside same page). Before doing the tests, login as USERNAME with password PASSWORD. E.g. interesting values for USERNAME are "guest" or "admin". :return: empty list in case of no problems, otherwise list of error messages that may have been encountered during processing of page. """ from invenio.utils.w3c_validator import (w3c_validate, w3c_errors_to_str, CFG_TESTS_REQUIRE_HTML_VALIDATION) if require_validate_p is None: require_validate_p = CFG_TESTS_REQUIRE_HTML_VALIDATION try: import mechanize except ImportError: raise InvenioTestUtilsBrowserException( 'ERROR: Cannot import mechanize.') if '--w3c-validate' in sys.argv: require_validate_p = True sys.stderr.write('Required validation\n') error_messages = [] try: browser = get_authenticated_mechanize_browser(username, password) try: browser.open(url) except mechanize.HTTPError as msg: if msg.code != 401: raise msg error_messages.append('ERROR: Page %s (login %s) not accessible. ' '%s' % (url, username, msg)) url_body = browser.response().read() # now test for EXPECTED_TEXT: # first normalize expected_text if isinstance(expected_text, str): expected_texts = [expected_text] else: expected_texts = expected_text # then test for cur_expected_text in expected_texts: try: url_body.index(cur_expected_text) except ValueError: raise InvenioTestUtilsBrowserException( 'ERROR: Page %s (login %s) does not contain %s, but ' 'contains %s', (url, username, cur_expected_text, url_body)) # now test for UNEXPECTED_TEXT: # first normalize unexpected_text if isinstance(unexpected_text, str): if unexpected_text: unexpected_texts = [unexpected_text] else: unexpected_texts = [] else: unexpected_texts = unexpected_text # then test for cur_unexpected_text in unexpected_texts: try: url_body.index(cur_unexpected_text) raise InvenioTestUtilsBrowserException( 'ERROR: Page %s (login %s) contains %s.' % (url, username, cur_unexpected_text)) except ValueError: pass # now test for EXPECTED_LINK_TARGET and EXPECTED_LINK_LABEL: if expected_link_target or expected_link_label: # first normalize expected_link_target and expected_link_label if isinstance(expected_link_target, str) or \ expected_link_target is None: expected_link_targets = [expected_link_target] else: expected_link_targets = expected_link_target if isinstance(expected_link_label, str) or \ expected_link_label is None: expected_link_labels = [expected_link_label] else: expected_link_labels = expected_link_label max_links = max( len(expected_link_targets), len(expected_link_labels)) expected_link_labels = chain(expected_link_labels, repeat(None)) expected_link_targets = chain(expected_link_targets, repeat(None)) # then test for dummy in range(0, max_links): cur_expected_link_target = expected_link_targets.next() cur_expected_link_label = expected_link_labels.next() try: browser.find_link(url=cur_expected_link_target, text=cur_expected_link_label) except mechanize.LinkNotFoundError: raise InvenioTestUtilsBrowserException( 'ERROR: Page %s (login %s) does not contain link to %s' 'entitled %s.' % (url, username, cur_expected_link_target, cur_expected_link_label) ) # now test for validation if required if require_validate_p: valid_p, errors, warnings = w3c_validate(url_body) if not valid_p: error_text = 'ERROR: Page %s (login %s) does not ' \ 'validate:\n %s' % \ (url, username, w3c_errors_to_str( errors, warnings)) from invenio.config import CFG_LOGDIR open('%s/w3c-markup-validator.log' % CFG_LOGDIR, 'a').write(error_text) raise InvenioTestUtilsBrowserException(error_text) except InvenioTestUtilsBrowserException as msg: error_messages.append( 'ERROR: Page %s (login %s) led to an error: %s.' % (url, username, msg)) try: # logout after tests: from invenio.config import CFG_SITE_SECURE_URL browser.open(CFG_SITE_SECURE_URL + "/youraccount/logout") browser.response().read() browser.close() except UnboundLocalError: pass except mechanize.HTTPError: # Raises 401 if you were not logged in before. pass if CFG_TESTUTILS_VERBOSE >= 9: print("%s test_web_page_content(), tested page `%s', login `%s', " "expected text `%s', errors `%s'." % (time.strftime("%Y-%m-%d %H:%M:%S -->", time.localtime()), url, username, expected_text, ",".join(error_messages))) return error_messages def merge_error_messages(error_messages): """Merge the error messages into a print-friendly version. If the ERROR_MESSAGES list is non-empty, merge them and return nicely formatted string suitable for printing. Otherwise return empty string. """ out = "" if error_messages: out = "\n*** " + "\n*** ".join(error_messages) return out # @nottest def build_and_run_unit_test_suite(): """Build and run the unit tests. Detect all Invenio modules with names ending by '*_unit_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-unit-tests'. """ from invenio.config import CFG_PYLIBDIR from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_unit_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = [test_modules_map[name] for name in test_modules_map] broken_tests = test_modules_map.get_broken_plugins() broken_unit_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_unit_tests: warn("Broken unit tests suites found: %s" % ', '.join(broken_unit_tests)) complete_suite = unittest.TestSuite(test_modules) runner = unittest.TextTestRunner(descriptions=False, verbosity=2) return runner.run(complete_suite).wasSuccessful() -# @nottest -def build_and_run_js_unit_test_suite(): - """Build and run the JavaScript unit tests. - - Init the JsTestDriver server, detect all Invenio JavaScript files with - names ending by '*_tests.js' and run them. - Called by 'inveniocfg --run-js-unit-tests'. - """ - from invenio.config import CFG_PREFIX, CFG_WEBDIR, CFG_JSTESTDRIVER_PORT - - def _server_init(server_process): - """Init JsTestDriver server and check if it succedeed.""" - output_success = "Finished action run" - output_error = "Server failed" - read_timeout = 30 - - start_time = time.time() - elapsed_time = 0 - while 1: - stdout_line = server_process.stdout.readline() - if output_success in stdout_line: - print('* JsTestDriver server ready\n') - return True - elif output_error in stdout_line or elapsed_time > read_timeout: - print('* ! JsTestDriver server init failed\n') - print(server_process.stdout.read()) - return False - elapsed_time = time.time() - start_time - - def _find_and_run_js_test_files(): - """Find and run all the JavaScript files. - - Find all JS files installed in Invenio lib directory and run - them on the JsTestDriver server - """ - from invenio.utils.shell import run_shell_command - errors_found = 0 - for candidate in os.listdir(CFG_WEBDIR + "/js"): - base, ext = os.path.splitext(candidate) - - if ext != '.js' or not base.endswith('_tests'): - continue - - print("Found test file %s. Running tests... " % (base + ext)) - exitcode_, stdout, stderr_ = run_shell_command( - cmd="java -jar %s/JsTestDriver.jar --config %s --tests all" % - (CFG_PREFIX + "/lib/java/js-test-driver", - CFG_WEBDIR + "/js/" + base + '.conf')) - print(stdout) - if "Fails: 0" not in stdout: - errors_found += 1 - print(errors_found) - return errors_found - - print("Going to start JsTestDriver server...") - server_process = subprocess.Popen([ - "java", - "-jar", - "%s/JsTestDriver.jar" % (CFG_PREFIX + "/lib/java/js-test-driver"), - "--runnerMode", "INFO", - "--port", - "%d" % CFG_JSTESTDRIVER_PORT], - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT - ) - - try: - from invenio.config import CFG_SITE_URL - if not _server_init(server_process): - # There was an error initialising server - return 1 - - print("Now you can capture the browsers where you would " - "like to run the tests by opening the following url:\n" - "%s:%d/capture \n" % (CFG_SITE_URL, CFG_JSTESTDRIVER_PORT)) - - print("Press enter when you are ready to run tests") - raw_input() - - exitcode = _find_and_run_js_test_files() - finally: - server_process.kill() - - return exitcode - - # @nottest def build_and_run_regression_test_suite(): """Build and run the regression tests. Detect all Invenio modules with names ending by '*_regression_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-regression-tests'. """ from invenio.config import CFG_PYLIBDIR from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_regression_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = test_modules_map.values() broken_tests = test_modules_map.get_broken_plugins() broken_regression_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_regression_tests: warn("Broken regression tests suites found: %s" % ', '.join(broken_regression_tests)) warn_user_about_tests() complete_suite = unittest.TestSuite(test_modules) runner = unittest.TextTestRunner(descriptions=False, verbosity=2) return runner.run(complete_suite).wasSuccessful() # @nottest def build_and_run_web_test_suite(): """Build and run the web tests. Detect all Invenio modules with names ending by '*_web_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-web-tests'. """ from invenio.config import CFG_PYLIBDIR from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_web_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = test_modules_map.values() broken_tests = test_modules_map.get_broken_plugins() broken_web_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_web_tests: warn("Broken web tests suites found: %s" % ', '.join(broken_web_tests)) warn_user_about_tests() complete_suite = unittest.TestSuite(test_modules) runner = unittest.TextTestRunner(descriptions=False, verbosity=2) return runner.run(complete_suite).wasSuccessful() class InvenioWebTestCase(InvenioTestCase): """ Base test case for the web environment. Helper library of useful web test functions for web tests creation. """ def setUp(self): """Initialization before tests.""" # Let's default to English locale profile = webdriver.FirefoxProfile() profile.set_preference('intl.accept_languages', 'en-us, en') profile.update_preferences() # the instance of Firefox WebDriver is created self.browser = webdriver.Firefox(profile) # list of errors self.errors = [] def tearDown(self): """Cleanup actions after tests.""" self.browser.quit() self.assertEqual([], self.errors) def find_element_by_name_with_timeout(self, element_name, timeout=30): """Find an element by name. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. :param element_name: name of the element to find :type element_name: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_name(element_name)) except Exception: raise InvenioWebTestCaseException(element=element_name) def find_element_by_link_text_with_timeout(self, element_link_text, timeout=30): """Find an element by link text. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. :param element_link_text: link text of the element to find :type element_link_text: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_link_text( element_link_text)) except Exception: raise InvenioWebTestCaseException(element=element_link_text) def find_element_by_partial_link_text_with_timeout( self, element_partial_link_text, timeout=30): """Find an element by partial link text. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. :param element_partial_link_text: partial link text of the element to find :type element_partial_link_text: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_partial_link_text( element_partial_link_text)) except Exception: raise InvenioWebTestCaseException( element=element_partial_link_text) def find_element_by_id_with_timeout(self, element_id, timeout=30, text=""): """Find an element by id. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. If the parameter text is provided, the function waits until the element is found and its content is equal to the given text. If the element's text is not equal to the given text an exception will be raised and the result of this comparison will be stored in the errors list **NOTE:** Currently this is used to wait for an element's text to be refreshed using JavaScript :param element_id: id of the element to find :type element_id: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int :param text: expected text inside the given element. :type text: string """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_id(element_id)) except Exception: raise InvenioWebTestCaseException(element=element_id) if text: q = self.browser.find_element_by_id(element_id) try: # if the element's text is not equal to the given text, an # exception will be raised WebDriverWait(self.browser, timeout).until( lambda driver: (driver.find_element_by_id(element_id) and q.text == text)) except Exception: # let's store the result of the comparison in the errors list try: self.assertEqual(q.text, text) except AssertionError as e: self.errors.append(str(e)) def find_element_by_xpath_with_timeout(self, element_xpath, timeout=30): """Find an element by xpath. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. :param element_xpath: xpath of the element to find :type element_xpath: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_xpath(element_xpath)) except Exception: raise InvenioWebTestCaseException(element=element_xpath) def find_elements_by_class_name_with_timeout(self, element_class_name, timeout=30): """Find an element by class name. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. :param element_class_name: class name of the element to find :type element_class_name: string :param timeout: time in seconds before throwing an exception if the element is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.find_element_by_class_name( element_class_name)) except Exception: raise InvenioWebTestCaseException(element=element_class_name) def find_page_source_with_timeout(self, timeout=30): """Find the page source. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if the page source is loaded will return it in 0 - timeout seconds. :param timeout: time in seconds before throwing an exception if the page source is not found :type timeout: int """ try: WebDriverWait(self.browser, timeout).until( lambda driver: driver.page_source) except Exception: raise InvenioWebTestCaseException(element="page source") def login(self, username="guest", password="", force_ln='en', go_to_login_page=True): """Log in. :param username: the username (nickname or email) :type username: str :param password: the corresponding password :type password: str :param force_ln: if the arrival page doesn't use the corresponding language, then the browser will redirect to it. :type force_ln: str :param go_to_login_page: if True, look for login link on the page. Otherwise expect to be already on a page with the login form :type go_to_login_page: bool """ if go_to_login_page: if ("You can use your nickname or your email address to login." not in self.browser.page_source): if ("You are no longer recognized by our system" in self.browser.page_source): self.find_element_by_link_text_with_timeout("login here") self.browser.find_element_by_link_text( "login here").click() else: self.find_element_by_link_text_with_timeout("login") self.browser.find_element_by_link_text("login").click() self.find_element_by_name_with_timeout("nickname") self.browser.find_element_by_name("nickname").clear() self.fill_textbox(textbox_name="nickname", text=username) self.find_element_by_name_with_timeout("password") self.browser.find_element_by_name("password").clear() self.fill_textbox(textbox_name="password", text=password) self.find_element_by_name_with_timeout("action") self.browser.find_element_by_name("action").click() from invenio.config import CFG_SITE_NAME_INTL if force_ln and (CFG_SITE_NAME_INTL[force_ln] not in self.browser.page_source): splitted_url = list(urlsplit(self.browser.current_url)) query = cgi.parse_qs(splitted_url[3]) query.update({u'ln': unicode(force_ln)}) splitted_url[3] = urlencode(query) new_url = urlunsplit(splitted_url) self.browser.get(new_url) def logout(self): """Log out.""" self.find_element_by_link_text_with_timeout("logout") self.browser.find_element_by_link_text("logout").click() @nottest def element_value_test(self, element_name="", element_id="", expected_element_value="", unexpected_element_value="", in_form=True): """Check if the value in the given element is the expected value. :param element_name: name of the corresponding element in the form :type element_name: string :param element_id: id of the corresponding element in the form :type element_id: string :param expected_element_value: the expected element value :type expected_element_value: string :param unexpected_element_value: the unexpected element value :type unexpected_element_value: string :param in_form: depends on this parameter, the value of the given element is got in a different way. If it is True, the given element is a textbox or a textarea in a form. :type in_form: boolean """ if element_name: self.find_element_by_name_with_timeout(element_name) q = self.browser.find_element_by_name(element_name) elif element_id: self.find_element_by_id_with_timeout(element_id) q = self.browser.find_element_by_id(element_id) if unexpected_element_value: try: if in_form: self.assertNotEqual( q.get_attribute('value'), unexpected_element_value) else: self.assertNotEqual(q.text, unexpected_element_value) except AssertionError as e: self.errors.append(str(e)) if expected_element_value: try: if in_form: self.assertEqual( q.get_attribute('value'), expected_element_value) else: self.assertEqual(q.text, expected_element_value) except AssertionError as e: self.errors.append(str(e)) @nottest def page_source_test(self, expected_text="", unexpected_text=""): """Check if the current page contains (or not) the expected text(s). The expected text (unexpected text) can also be a link. The expected text (unexpected text) can be a list of strings in order to check multiple values inside same page :param expected_text: the expected text :type expected_text: string or list of strings :param unexpected_text: the unexpected text :type unexpected_text: string or list of strings """ self.find_page_source_with_timeout() if unexpected_text: if isinstance(unexpected_text, str): unexpected_texts = [unexpected_text] else: unexpected_texts = unexpected_text for unexpected_text in unexpected_texts: try: self.assertEqual( -1, self.browser.page_source.find(unexpected_text)) except AssertionError as e: self.errors.append(str(e)) if expected_text: if isinstance(expected_text, str): expected_texts = [expected_text] else: expected_texts = expected_text for expected_text in expected_texts: try: self.assertNotEqual( -1, self.browser.page_source.find(expected_text)) except AssertionError as e: self.errors.append(str(e)) def choose_selectbox_option_by_label(self, selectbox_name="", selectbox_id="", label=""): """Select the option at the given label in the select box. :param selectbox_name: the name of the corresponding select box in the form :type selectbox_name: string :param selectbox_id: the id of the corresponding select box in the form :type selectbox_id: string :param label: the option at this label will be selected :type label: string """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") for option in options: if option.text == label: option.click() break def choose_selectbox_option_by_index(self, selectbox_name="", selectbox_id="", index=""): """Select the option at the given index in the select box. :param selectbox_name: the name of the corresponding select box in the form :type selectbox_name: string :param selectbox_id: the id of the corresponding select box in the form :type selectbox_id: string :param index: the option at this index will be selected :type index: int """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") options[int(index)].click() def choose_selectbox_option_by_value(self, selectbox_name="", selectbox_id="", value=""): """Select the option at the given value in the select box. :param selectbox_name: the name of the corresponding select box in the form :type selectbox_id: string :param selectbox_id: the id of the corresponding select box in the form :type selectbox_id: string :param value: the option at this value will be selected :type value: string """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") for option in options: if option.get_attribute('value') == value: option.click() break def fill_textbox(self, textbox_name="", textbox_id="", text=""): """Fill in the input textbox or textarea with the given text. :param textbox_name: the name of the corresponding textbox or text area in the form :type textbox_name: string :param textbox_id: the id of the corresponding textbox or text area in the form :type textbox_id: string :param text: the information that the user wants to send :type text: string """ if textbox_name: self.find_element_by_name_with_timeout(textbox_name) textbox = self.browser.find_element_by_name(textbox_name) elif textbox_id: self.find_element_by_id_with_timeout(textbox_id) textbox = self.browser.find_element_by_id(textbox_id) textbox.send_keys(text) def handle_popup_dialog(self): """Access the alert after triggering an action that opens a popup.""" try: alert = self.browser.switch_to_alert() alert.accept() except Exception: pass class InvenioWebTestCaseException(Exception): """Exception for the web tests. This exception is thrown if the element we are looking for is not found after a set time period. The element is not found because the page needs more time to be fully loaded. To avoid this exception, we should increment the time period for that element in the corresponding function. See also: find_element_by_name_with_timeout() find_element_by_link_text_with_timeout() find_element_by_partial_link_text_with_timeout() find_element_by_id_with_timeout() find_element_by_xpath_with_timeout() find_elements_by_class_name_with_timeout() find_page_source_with_timeout() """ def __init__(self, element): """Initialisation.""" self.element = element self.message = "Time for finding the element '{0}' has expired".format( self.element) def __str__(self): """String representation.""" return repr(self.message) # @nottest def build_and_run_flask_test_suite(): """Build and run the flask tests. Detect all Invenio modules with names ending by '*_flask_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-flask-tests'. """ test_modules = [] import invenio for candidate in os.listdir(os.path.dirname(invenio.__file__)): base, ext = os.path.splitext(candidate) if ext != '.py' or not base.endswith('_flask_tests'): continue module = __import__( 'invenio.' + base, globals(), locals(), ['TEST_SUITE']) test_modules.append(module.TEST_SUITE) # FIXME create warning about tested databases warn_user_about_tests() complete_suite = unittest.TestSuite(test_modules) run_test_suite(complete_suite) def iter_suites(packages=None): """Yield all testsuites.""" from werkzeug.utils import import_string, find_modules from flask_registry import ModuleAutoDiscoveryRegistry, \ ImportPathRegistry app = create_app() if packages is None: testsuite = ModuleAutoDiscoveryRegistry('testsuite', app=app) from invenio import testsuite as testsuite_invenio from invenio.base import testsuite as testsuite_base from invenio.celery import testsuite as testsuite_celery testsuite.register(testsuite_invenio) testsuite.register(testsuite_base) testsuite.register(testsuite_celery) else: exclude = map(lambda x: x + '.testsuite', app.config.get('PACKAGES_EXCLUDE', [])) testsuite = ImportPathRegistry(initial=packages, exclude=exclude, load_modules=True) for package in testsuite: for name in find_modules(package.__name__): module = import_string(name) if not module.__name__.split('.')[-1].startswith('test_'): continue if hasattr(module, 'TEST_SUITE'): yield module.TEST_SUITE else: app.logger.warning( "%s: No test suite defined." % module.__name__) def suite(): """Create the testsuite that has all the tests.""" suite = unittest.TestSuite() for other_suite in iter_suites(): suite.addTest(other_suite) return suite def main(): """Run the testsuite as command line application.""" try: unittest.main(defaultTest='suite') except Exception as e: print(('Error: %s' % e)) diff --git a/invenio/testsuite/helper.py b/invenio/testsuite/helper.py index c9acd9907..fc5dff473 100644 --- a/invenio/testsuite/helper.py +++ b/invenio/testsuite/helper.py @@ -1,1007 +1,932 @@ # This file is part of Invenio. -# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 CERN. +# Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. # pylint: disable=E1102 """ Helper functions for building and running test suites. """ __revision__ = "$Id$" CFG_TESTUTILS_VERBOSE = 1 import os import sys import time import unittest import cgi import subprocess from warnings import warn from functools import wraps from urlparse import urlsplit, urlunsplit from urllib import urlencode from itertools import chain, repeat try: from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait except ImportError: # web tests will not be available, but unit and regression tests will: pass from invenio.config import (CFG_SITE_URL, CFG_SITE_SECURE_URL, CFG_LOGDIR, CFG_SITE_NAME_INTL, CFG_PYLIBDIR, CFG_JSTESTDRIVER_PORT, CFG_WEBDIR, - CFG_PREFIX, CFG_BASE_URL) from invenio.utils.w3c_validator import (w3c_validate, w3c_errors_to_str, CFG_TESTS_REQUIRE_HTML_VALIDATION) try: from nose.tools import nottest except ImportError: def nottest(f): """Helper decorator to mark a function as not to be tested by nose.""" f.__test__ = False return f @nottest def warn_user_about_tests(test_suite_type='regression'): """ Display a standard warning about running tests that might modify user data, and wait for user confirmation, unless --yes-i-know was specified in the comman line. """ # Provide a command line option to avoid having to type the # confirmation every time during development. if '--yes-i-know' in sys.argv: return if test_suite_type == 'web': sys.stderr.write("""\ ********************************************************************** ** ** ** A B O U T T H E W E B T E S T S U I T E ** ** ** ** The web test suite will be launched in Firefox. You must have ** ** the Selenium IDE extension installed to be able to run the web ** ** test suite. If you do, please check out the results of the web ** ** test suite in the Selenium IDE window. ** ** ** ********************************************************************** """) sys.stderr.write("""\ ********************************************************************** ** ** ** I M P O R T A N T W A R N I N G ** ** ** ** The %s test suite needs to be run on a clean demo site ** ** that you can obtain by doing: ** ** ** ** $ inveniocfg --drop-demo-site \ ** ** --create-demo-site \ ** ** --load-demo-records ** ** ** ** Note that DOING THE ABOVE WILL ERASE YOUR ENTIRE DATABASE. ** ** ** ** In addition, due to the write nature of some of the tests, ** ** the demo DATABASE will be ALTERED WITH JUNK DATA, so that ** ** it is recommended to rebuild the demo site anew afterwards. ** ** ** ********************************************************************** Please confirm by typing 'Yes, I know!': """ % test_suite_type) answer = raw_input('') if answer != 'Yes, I know!': sys.stderr.write("Aborted.\n") raise SystemExit(0) return @nottest def make_test_suite(*test_cases): """ Build up a test suite given separate test cases""" return unittest.TestSuite([unittest.makeSuite(case, 'test') for case in test_cases]) @nottest def run_test_suite(testsuite, warn_user=False): """ Convenience function to embed in test suites. Run given testsuite and eventually ask for confirmation of warn_user is True. """ if warn_user: warn_user_about_tests() res = unittest.TextTestRunner(verbosity=2).run(testsuite) return res.wasSuccessful() def make_url(path, **kargs): """ Helper to generate an absolute invenio URL with query arguments""" url = CFG_SITE_URL + path if kargs: url += '?' + urlencode(kargs, doseq=True) return url def make_surl(path, **kargs): """ Helper to generate an absolute invenio Secure URL with query arguments""" url = CFG_SITE_SECURE_URL + path if kargs: url += '?' + urlencode(kargs, doseq=True) return url def make_rurl(path, **kargs): """ Helper to generate an relative invenio URL with query arguments""" url = CFG_BASE_URL + path if kargs: url += '?' + urlencode(kargs, doseq=True) return url class InvenioTestUtilsBrowserException(Exception): """Helper exception for the regression test suite browser.""" pass @nottest def test_web_page_existence(url): """ Test whether URL exists and is well accessible. Return True or raise exception in case of problems. """ import mechanize browser = mechanize.Browser() try: browser.open(url) except: raise return True def get_authenticated_mechanize_browser(username="guest", password=""): """ Return an instance of a mechanize browser already authenticated to Invenio """ try: import mechanize except ImportError: raise InvenioTestUtilsBrowserException('ERROR: Cannot import mechanize.') browser = mechanize.Browser() browser.set_handle_robots(False) # ignore robots.txt, since we test gently if username == "guest": return browser browser.open(CFG_SITE_SECURE_URL + "/youraccount/login") browser.select_form(nr=0) browser['p_un'] = username browser['p_pw'] = password browser.submit() username_account_page_body = browser.response().read() try: username_account_page_body.index("You are logged in as %s." % username) except ValueError: raise InvenioTestUtilsBrowserException('ERROR: Cannot login as %s.' % username) return browser @nottest def test_web_page_content(url, username="guest", password="", expected_text="", unexpected_text="", expected_link_target=None, expected_link_label=None, require_validate_p=CFG_TESTS_REQUIRE_HTML_VALIDATION): """Test whether web page URL as seen by user USERNAME contains text EXPECTED_TEXT and, eventually, contains a link to EXPECTED_LINK_TARGET (if set) labelled EXPECTED_LINK_LABEL (if set). The EXPECTED_TEXT is checked via substring matching, the EXPECTED_LINK_TARGET and EXPECTED_LINK_LABEL via exact string matching. EXPECTED_TEXT, EXPECTED_LINK_LABEL and EXPECTED_LINK_TARGET can either be strings or list of strings (in order to check multiple values inside same page). Before doing the tests, login as USERNAME with password PASSWORD. E.g. interesting values for USERNAME are "guest" or "admin". Return empty list in case of no problems, otherwise list of error messages that may have been encountered during processing of page. """ try: import mechanize except ImportError: raise InvenioTestUtilsBrowserException('ERROR: Cannot import mechanize.') if '--w3c-validate' in sys.argv: require_validate_p = True sys.stderr.write('Required validation\n') error_messages = [] try: browser = get_authenticated_mechanize_browser(username, password) browser.open(url) url_body = browser.response().read() # now test for EXPECTED_TEXT: # first normalize expected_text if isinstance(expected_text, str): expected_texts = [expected_text] else: expected_texts = expected_text # then test for cur_expected_text in expected_texts: try: url_body.index(cur_expected_text) except ValueError: raise InvenioTestUtilsBrowserException, \ 'ERROR: Page %s (login %s) does not contain %s, but contains %s' % \ (url, username, cur_expected_text, url_body) # now test for UNEXPECTED_TEXT: # first normalize unexpected_text if isinstance(unexpected_text, str): if unexpected_text: unexpected_texts = [unexpected_text] else: unexpected_texts = [] else: unexpected_texts = unexpected_text # then test for cur_unexpected_text in unexpected_texts: try: url_body.index(cur_unexpected_text) raise InvenioTestUtilsBrowserException, \ 'ERROR: Page %s (login %s) contains %s.' % \ (url, username, cur_unexpected_text) except ValueError: pass # now test for EXPECTED_LINK_TARGET and EXPECTED_LINK_LABEL: if expected_link_target or expected_link_label: # first normalize expected_link_target and expected_link_label if isinstance(expected_link_target, str) or \ expected_link_target is None: expected_link_targets = [expected_link_target] else: expected_link_targets = expected_link_target if isinstance(expected_link_label, str) or \ expected_link_label is None: expected_link_labels = [expected_link_label] else: expected_link_labels = expected_link_label max_links = max(len(expected_link_targets), len(expected_link_labels)) expected_link_labels = chain(expected_link_labels, repeat(None)) expected_link_targets = chain(expected_link_targets, repeat(None)) # then test for dummy in range(0, max_links): cur_expected_link_target = expected_link_targets.next() cur_expected_link_label = expected_link_labels.next() try: browser.find_link(url=cur_expected_link_target, text=cur_expected_link_label) except mechanize.LinkNotFoundError: raise InvenioTestUtilsBrowserException, \ 'ERROR: Page %s (login %s) does not contain link to %s entitled %s.' % \ (url, username, cur_expected_link_target, cur_expected_link_label) # now test for validation if required if require_validate_p: valid_p, errors, warnings = w3c_validate(url_body) if not valid_p: error_text = 'ERROR: Page %s (login %s) does not validate:\n %s' % \ (url, username, w3c_errors_to_str(errors, warnings)) open('%s/w3c-markup-validator.log' % CFG_LOGDIR, 'a').write(error_text) raise InvenioTestUtilsBrowserException, error_text except mechanize.HTTPError, msg: error_messages.append('ERROR: Page %s (login %s) not accessible. %s' % \ (url, username, msg)) except InvenioTestUtilsBrowserException, msg: error_messages.append('ERROR: Page %s (login %s) led to an error: %s.' % \ (url, username, msg)) try: # logout after tests: browser.open(CFG_SITE_SECURE_URL + "/youraccount/logout") browser.response().read() browser.close() except UnboundLocalError: pass if CFG_TESTUTILS_VERBOSE >= 9: print "%s test_web_page_content(), tested page `%s', login `%s', expected text `%s', errors `%s'." % \ (time.strftime("%Y-%m-%d %H:%M:%S -->", time.localtime()), url, username, expected_text, ",".join(error_messages)) return error_messages def merge_error_messages(error_messages): """If the ERROR_MESSAGES list is non-empty, merge them and return nicely formatted string suitable for printing. Otherwise return empty string. """ out = "" if error_messages: out = "\n*** " + "\n*** ".join(error_messages) return out @nottest def build_and_run_unit_test_suite(): """ Detect all Invenio modules with names ending by '*_unit_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-unit-tests'. """ from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_unit_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = [test_modules_map[name] for name in test_modules_map] broken_tests = test_modules_map.get_broken_plugins() broken_unit_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_unit_tests: warn("Broken unit tests suites found: %s" % ', '.join(broken_unit_tests)) complete_suite = unittest.TestSuite(test_modules) res = unittest.TextTestRunner(verbosity=2).run(complete_suite) return res.wasSuccessful() -@nottest -def build_and_run_js_unit_test_suite(): - """ - Init the JsTestDriver server, detect all Invenio JavaScript files with - names ending by '*_tests.js' and run them. - Called by 'inveniocfg --run-js-unit-tests'. - """ - def _server_init(server_process): - """ - Init JsTestDriver server and check if it succedeed - """ - output_success = "Finished action run" - output_error = "Server failed" - read_timeout = 30 - - start_time = time.time() - elapsed_time = 0 - while 1: - stdout_line = server_process.stdout.readline() - if output_success in stdout_line: - print '* JsTestDriver server ready\n' - return True - elif output_error in stdout_line or elapsed_time > read_timeout: - print '* ! JsTestDriver server init failed\n' - print server_process.stdout.read() - return False - elapsed_time = time.time() - start_time - - def _find_and_run_js_test_files(): - """ - Find all JS files installed in Invenio lib directory and run - them on the JsTestDriver server - """ - from invenio.shellutils import run_shell_command - errors_found = 0 - for candidate in os.listdir(CFG_WEBDIR + "/js"): - base, ext = os.path.splitext(candidate) - - if ext != '.js' or not base.endswith('_tests'): - continue - - print "Found test file %s. Running tests... " % (base + ext) - dummy_current_exitcode, cmd_stdout, dummy_err_msg = run_shell_command(cmd="java -jar %s/JsTestDriver.jar --config %s --tests all" % \ - (CFG_PREFIX + "/lib/java/js-test-driver", CFG_WEBDIR + "/js/" + base + '.conf')) - print cmd_stdout - if "Fails: 0" not in cmd_stdout: - errors_found += 1 - print errors_found - return errors_found - - print "Going to start JsTestDriver server..." - server_process = subprocess.Popen(["java", "-jar", - "%s/JsTestDriver.jar" % (CFG_PREFIX + "/lib/java/js-test-driver"), "--runnerMode", "INFO", - "--port", "%d" % CFG_JSTESTDRIVER_PORT], - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - - try: - if not _server_init(server_process): - # There was an error initialising server - return 1 - - print "Now you can capture the browsers where you would " \ - "like to run the tests by opening the following url:\n" \ - "%s:%d/capture \n" % (CFG_SITE_URL, CFG_JSTESTDRIVER_PORT) - - print "Press enter when you are ready to run tests" - raw_input() - - exitcode = _find_and_run_js_test_files() - finally: - server_process.kill() - - return exitcode - @nottest def build_and_run_regression_test_suite(): """ Detect all Invenio modules with names ending by '*_regression_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-regression-tests'. """ from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_regression_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = test_modules_map.values() broken_tests = test_modules_map.get_broken_plugins() broken_regression_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_regression_tests: warn("Broken regression tests suites found: %s" % ', '.join(broken_regression_tests)) warn_user_about_tests() complete_suite = unittest.TestSuite(test_modules) res = unittest.TextTestRunner(verbosity=2).run(complete_suite) return res.wasSuccessful() @nottest def build_and_run_web_test_suite(): """ Detect all Invenio modules with names ending by '*_web_tests.py', build a complete test suite of them, and run it. Called by 'inveniocfg --run-web-tests'. """ from invenio.pluginutils import PluginContainer test_modules_map = PluginContainer( os.path.join(CFG_PYLIBDIR, 'invenio', '*_web_tests.py'), lambda plugin_name, plugin_code: getattr(plugin_code, "TEST_SUITE")) test_modules = test_modules_map.values() broken_tests = test_modules_map.get_broken_plugins() broken_web_tests = ['%s (reason: %s)' % (name, broken_tests[name][1]) for name in broken_tests] if broken_web_tests: warn("Broken web tests suites found: %s" % ', '.join(broken_web_tests)) warn_user_about_tests() complete_suite = unittest.TestSuite(test_modules) res = unittest.TextTestRunner(verbosity=2).run(complete_suite) return res.wasSuccessful() class InvenioTestCase(unittest.TestCase): "Invenio Test Case class." pass try: InvenioTestCase.assertMultiLineEqual except AttributeError: InvenioTestCase.assertMultiLineEqual = InvenioTestCase.assertEqual class InvenioWebTestCase(unittest.TestCase): """ Helper library of useful web test functions for web tests creation. """ def setUp(self): """Initialization before tests.""" ## Let's default to English locale profile = webdriver.FirefoxProfile() profile.set_preference('intl.accept_languages', 'en-us, en') profile.update_preferences() # the instance of Firefox WebDriver is created self.browser = webdriver.Firefox(profile) # list of errors self.errors = [] def tearDown(self): """Cleanup actions after tests.""" self.browser.quit() self.assertEqual([], self.errors) def find_element_by_name_with_timeout(self, element_name, timeout=30): """ Find an element by name. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param element_name: name of the element to find @type element_name: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_name(element_name)) except: raise InvenioWebTestCaseException(element=element_name) def find_element_by_link_text_with_timeout(self, element_link_text, timeout=30): """ Find an element by link text. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param element_link_text: link text of the element to find @type element_link_text: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_link_text(element_link_text)) except: raise InvenioWebTestCaseException(element=element_link_text) def find_element_by_partial_link_text_with_timeout(self, element_partial_link_text, timeout=30): """ Find an element by partial link text. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param element_partial_link_text: partial link text of the element to find @type element_partial_link_text: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_partial_link_text(element_partial_link_text)) except: raise InvenioWebTestCaseException(element=element_partial_link_text) def find_element_by_id_with_timeout(self, element_id, timeout=30, text=""): """ Find an element by id. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. If the parameter text is provided, the function waits until the element is found and its content is equal to the given text. If the element's text is not equal to the given text an exception will be raised and the result of this comparison will be stored in the errors list #NOTE: Currently this is used to wait for an element's text to be refreshed using JavaScript @param element_id: id of the element to find @type element_id: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int @param text: expected text inside the given element. @type text: string """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_id(element_id)) except: raise InvenioWebTestCaseException(element=element_id) if text: q = self.browser.find_element_by_id(element_id) try: # if the element's text is not equal to the given text, an exception will be raised WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_id(element_id) and q.text==text) except: # let's store the result of the comparison in the errors list try: self.assertEqual(q.text, text) except AssertionError, e: self.errors.append(str(e)) def find_element_by_xpath_with_timeout(self, element_xpath, timeout=30): """ Find an element by xpath. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param element_xpath: xpath of the element to find @type element_xpath: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_xpath(element_xpath)) except: raise InvenioWebTestCaseException(element=element_xpath) def find_elements_by_class_name_with_timeout(self, elements_class_name, timeout=30): """ Find elements by class name. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param elements_class_name: class name of the elements to find @type elements_class_name: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_elements_by_class_name(elements_class_name)) except: raise InvenioWebTestCaseException(element=elements_class_name) def find_element_by_class_name_with_timeout(self, element_class_name, timeout=30): """ Find an element by class name. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if it finds the element will return it in 0 - timeout seconds. @param element_class_name: class name of the element to find @type element_class_name: string @param timeout: time in seconds before throwing an exception if the element is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.find_element_by_class_name(element_class_name)) except: raise InvenioWebTestCaseException(element=element_class_name) def find_page_source_with_timeout(self, timeout=30): """ Find the page source. This waits up to 'timeout' seconds before throwing an InvenioWebTestCaseException or if the page source is loaded will return it in 0 - timeout seconds. @param timeout: time in seconds before throwing an exception if the page source is not found @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda driver: driver.page_source) except: raise InvenioWebTestCaseException(element="page source") def wait_element_displayed_with_timeout(self, element, timeout=30): """ Wait until the given element is displayed, or timeout is reached @param element: object we want to wait to be displayed @type element: selenium.webdriver.remote.webelement.WebElement @param timeout: time in seconds before throwing an exception if element does not become visible @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda x: element.is_displayed()) except: raise InvenioWebTestCaseException(element=element) def wait_element_hidden_with_timeout(self, element, timeout=30): """ Wait until the given element is hidden, or timeout is reached @param element: object we want to wait to be hidden @type element: selenium.webdriver.remote.webelement.WebElement @param timeout: time in seconds before throwing an exception if element does not become hidden @type timeout: int """ try: WebDriverWait(self.browser, timeout).until(lambda x: not element.is_displayed()) except: raise InvenioWebTestCaseException(element=element) def login(self, username="guest", password="", force_ln='en', go_to_login_page=True): """ Login function @param username: the username (nickname or email) @type username: string @param password: the corresponding password @type password: string @param force_ln: if the arrival page doesn't use the corresponding language, then the browser will redirect to it. @type force_ln: string @param go_to_login_page: if True, look for login link on the page. Otherwise expect to be already on a page with the login form @type go_to_login_page: bool """ if go_to_login_page: if not "You can use your nickname or your email address to login." in self.browser.page_source: if "You are no longer recognized by our system" in self.browser.page_source: self.find_element_by_link_text_with_timeout("login here") self.browser.find_element_by_link_text("login here").click() else: self.find_element_by_link_text_with_timeout("login") self.browser.find_element_by_link_text("login").click() self.find_element_by_name_with_timeout("p_un") self.browser.find_element_by_name("p_un").clear() self.fill_textbox(textbox_name="p_un", text=username) self.find_element_by_name_with_timeout("p_pw") self.browser.find_element_by_name("p_pw").clear() self.fill_textbox(textbox_name="p_pw", text=password) self.find_element_by_name_with_timeout("action") self.browser.find_element_by_name("action").click() if force_ln and CFG_SITE_NAME_INTL[force_ln] not in self.browser.page_source: splitted_url = list(urlsplit(self.browser.current_url)) query = cgi.parse_qs(splitted_url[3]) query.update({u'ln': unicode(force_ln)}) splitted_url[3] = urlencode(query) new_url = urlunsplit(splitted_url) self.browser.get(new_url) def logout(self): """ Logout function """ self.find_element_by_link_text_with_timeout("logout") self.browser.find_element_by_link_text("logout").click() @nottest def element_value_test(self, element_name="", element_id="", \ expected_element_value="", unexpected_element_value="", in_form=True, exact_match=True): """ Function to check if the value in the given element is the expected (unexpected) value or not @param element_name: name of the corresponding element in the form @type element_name: string @param element_id: id of the corresponding element in the form @type element_id: string @param expected_element_value: the expected element value @type expected_element_value: string @param unexpected_element_value: the unexpected element value @type unexpected_element_value: string @param in_form: depends on this parameter, the value of the given element is got in a different way. If it is True, the given element is a textbox or a textarea in a form. @type in_form: boolean """ if element_name: self.find_element_by_name_with_timeout(element_name) q = self.browser.find_element_by_name(element_name) elif element_id: self.find_element_by_id_with_timeout(element_id) q = self.browser.find_element_by_id(element_id) if unexpected_element_value: try: if in_form: self.assertNotEqual(q.get_attribute('value'), unexpected_element_value) else: self.assertNotEqual(q.text, unexpected_element_value) except AssertionError, e: self.errors.append(str(e)) if expected_element_value: try: if in_form: if exact_match: self.assertEqual(q.get_attribute('value'), expected_element_value) else: self.assertNotEqual(-1, q.get_attribute('value').find(expected_element_value)) else: if exact_match: self.assertEqual(q.text, expected_element_value) else: self.assertNotEqual(-1, q.text.find(expected_element_value)) except AssertionError, e: self.errors.append(str(e)) @nottest def page_source_test(self, expected_text="", unexpected_text=""): """ Function to check if the current page contains the expected text (unexpected text) or not. The expected text (unexpected text) can also be a link. The expected text (unexpected text) can be a list of strings in order to check multiple values inside same page @param expected_text: the expected text @type expected_text: string or list of strings @param unexpected_text: the unexpected text @type unexpected_text: string or list of strings """ self.find_page_source_with_timeout() if unexpected_text: if isinstance(unexpected_text, str): unexpected_texts = [unexpected_text] else: unexpected_texts = unexpected_text for unexpected_text in unexpected_texts: try: self.assertEqual(-1, self.browser.page_source.find(unexpected_text)) except AssertionError, e: self.errors.append(str(e)) if expected_text: if isinstance(expected_text, str): expected_texts = [expected_text] else: expected_texts = expected_text for expected_text in expected_texts: try: self.assertNotEqual(-1, self.browser.page_source.find(expected_text)) except AssertionError, e: self.errors.append(str(e)) def choose_selectbox_option_by_label(self, selectbox_name="", selectbox_id="", label=""): """ Select the option at the given label in the corresponding select box @param selectbox_name: the name of the corresponding select box in the form @type selectbox_name: string @param selectbox_id: the id of the corresponding select box in the form @type selectbox_id: string @param label: the option at this label will be selected @type label: string """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") for option in options: if option.text == label: option.click() break def choose_selectbox_option_by_index(self, selectbox_name="", selectbox_id="", index=""): """ Select the option at the given index in the corresponding select box @param selectbox_name: the name of the corresponding select box in the form @type selectbox_name: string @param selectbox_id: the id of the corresponding select box in the form @type selectbox_id: string @param index: the option at this index will be selected @type index: int """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") options[int(index)].click() def choose_selectbox_option_by_value(self, selectbox_name="", selectbox_id="", value=""): """ Select the option at the given value in the corresponding select box @param selectbox_name: the name of the corresponding select box in the form @type selectbox_id: string @param selectbox_id: the id of the corresponding select box in the form @type selectbox_id: string @param value: the option at this value will be selected @type value: string """ if selectbox_name: self.find_element_by_name_with_timeout(selectbox_name) selectbox = self.browser.find_element_by_name(selectbox_name) elif selectbox_id: self.find_element_by_id_with_timeout(selectbox_id) selectbox = self.browser.find_element_by_id(selectbox_id) options = selectbox.find_elements_by_tag_name("option") for option in options: if option.get_attribute('value') == value: option.click() break def fill_textbox(self, textbox_name="", textbox_id="", text=""): """ Fill in the input textbox or textarea with the given text @param textbox_name: the name of the corresponding textbox or text area in the form @type textbox_name: string @param textbox_id: the id of the corresponding textbox or text area in the form @type textbox_id: string @param text: the information that the user wants to send @type text: string """ if textbox_name: self.find_element_by_name_with_timeout(textbox_name) textbox = self.browser.find_element_by_name(textbox_name) elif textbox_id: self.find_element_by_id_with_timeout(textbox_id) textbox = self.browser.find_element_by_id(textbox_id) textbox.send_keys(text) def handle_popup_dialog(self): """ Access the alert after triggering an action that opens a popup. """ try: alert = self.browser.switch_to_alert() alert.accept() except: pass class InvenioWebTestCaseException(Exception): """This exception is thrown if the element we are looking for is not found after a set time period. The element is not found because the page needs more time to be fully loaded. To avoid this exception, we should increment the time period for that element in the corresponding function. See also: find_element_by_name_with_timeout() find_element_by_link_text_with_timeout() find_element_by_partial_link_text_with_timeout() find_element_by_id_with_timeout() find_element_by_xpath_with_timeout() find_element_by_class_name_with_timeout() find_elements_by_class_name_with_timeout() find_page_source_with_timeout() """ def __init__(self, element): """Initialisation.""" self.element = element self.message = "Time for finding the element '%s' has expired" % self.element def __str__(self): """String representation.""" return repr(self.message) def failfast(method): @wraps(method) def inner(self, *args, **kw): self.stop() return method(self, *args, **kw) return inner def wrap_failfast(): """Makes it so unit tests will fail at the first error""" unittest.TestResult.addError = failfast(unittest.TestResult.addError) unittest.TestResult.addFailure = failfast(unittest.TestResult.addFailure) unittest.TestResult.addUnexpectedSuccess = failfast(unittest.TestResult.addUnexpectedSuccess) diff --git a/invenio/testsuite/test_utils_autodiscovery.py b/invenio/testsuite/test_utils_autodiscovery.py index a3c91e01d..e80289de6 100644 --- a/invenio/testsuite/test_utils_autodiscovery.py +++ b/invenio/testsuite/test_utils_autodiscovery.py @@ -1,53 +1,49 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2009, 2010, 2011, 2012, 2013 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """ Test unit for the miscutil/importutils module. """ from invenio.utils.autodiscovery import autodiscover_modules from invenio.testsuite import make_test_suite, run_test_suite, InvenioTestCase, nottest class AutodiscoveryTest(InvenioTestCase): """ Autodiscovery TestSuite. """ def test_autodiscover_modules(self): """autodiscover modules""" - modules = autodiscover_modules( - ['invenio.modules.formatter.format_elements'], - related_name_re='bfe_.+', ignore_exceptions=True) - assert(len(modules) > 10) modules = autodiscover_modules(['invenio.base'], related_name_re='config') assert(len(modules) == 1) assert(None not in modules) modules = autodiscover_modules(['invenio.not_an_existing_folder'], related_name_re='foo_.+') assert(len(modules) == 0) assert(None not in modules) modules = autodiscover_modules(['invenio.modules.formatter.format_elements'], related_name_re='not_an_existing_package_name_.+') assert(len(modules) == 0) assert(None not in modules) TEST_SUITE = make_test_suite(AutodiscoveryTest, ) if __name__ == "__main__": run_test_suite(TEST_SUITE) diff --git a/invenio/utils/datacite.py b/invenio/utils/datacite.py index 39f94b4f1..220e101de 100644 --- a/invenio/utils/datacite.py +++ b/invenio/utils/datacite.py @@ -1,176 +1,126 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Utilities for working with DataCite metadata.""" from __future__ import absolute_import import re import urllib2 -import warnings - -from datacite import DataCiteMDSClient -from datacite.errors import DataCiteBadRequestError, \ - DataCiteForbiddenError, \ - DataCiteGoneError, \ - DataCiteNoContentError, \ - DataCiteNotFoundError, \ - DataCitePreconditionError, \ - DataCiteRequestError, \ - DataCiteServerError, \ - DataCiteUnauthorizedError -from datacite.request import DataCiteRequest - -from invenio.base.globals import cfg -from invenio.utils.deprecation import RemovedInInvenio22Warning + from invenio.utils.xmlDict import ElementTree, XmlDictConfig __all__ = ( - 'DataCite', 'DataciteMetadata', 'DataCiteRequest', - 'DataCiteServerError', 'DataCiteRequestError', 'DataCiteNoContentError', - 'DataCiteBadRequestError', 'DataCiteUnauthorizedError', - 'DataCiteForbiddenError', 'DataCiteNotFoundError', 'DataCiteGoneError', - 'DataCitePreconditionError' + 'DataciteMetadata', ) -class DataCite(DataCiteMDSClient): - - """DataCite API wrapper. - - .. deprecated:: 2.0.2 - Use http://datacite.readthedocs.org/en/latest/ instead. - """ - - def __init__(self, username=None, password=None, url=None, prefix=None, - test_mode=None, api_ver="2"): - """Initialize API client. - - Compatibility layer on top of external DataCite API client. - """ - warnings.warn("Use of invenio.utils.datacite:DataCite is " - "deprecated in favor of " - "http://datacite.readthedocs.org/en/latest/.", - RemovedInInvenio22Warning) - - super(DataCite, self).__init__( - username=username or cfg.get('CFG_DATACITE_USERNAME', ''), - password=password or cfg.get('CFG_DATACITE_PASSWORD', ''), - url=url or cfg.get('CFG_DATACITE_URL', - 'https://mds.datacite.org/'), - prefix=prefix or cfg.get('CFG_DATACITE_DOI_PREFIX', '10.5072'), - test_mode=test_mode if test_mode is not None else cfg.get( - 'CFG_DATACITE_TESTMODE', False), - api_ver=api_ver or "2" - ) - - class DataciteMetadata(object): """Helper class for working with DataCite metadata.""" def __init__(self, doi): """Initialize object.""" self.url = "http://data.datacite.org/application/x-datacite+xml/" self.error = False try: data = urllib2.urlopen(self.url + doi).read() except urllib2.HTTPError: self.error = True if not self.error: # Clean the xml for parsing data = re.sub('<\?xml.*\?>', '', data, count=1) # Remove the resource tags data = re.sub('', '', data) self.data = '' + \ data[0:len(data) - 11] + '' self.root = ElementTree.XML(self.data) self.xml = XmlDictConfig(self.root) def get_creators(self, attribute='creatorName'): """Get DataCite creators.""" if 'creators' in self.xml: if isinstance(self.xml['creators']['creator'], list): return [c[attribute] for c in self.xml['creators']['creator']] else: return self.xml['creators']['creator'][attribute] return None def get_titles(self): """Get DataCite titles.""" if 'titles' in self.xml: return self.xml['titles']['title'] return None def get_publisher(self): """Get DataCite publisher.""" if 'publisher' in self.xml: return self.xml['publisher'] return None def get_dates(self): """Get DataCite dates.""" if 'dates' in self.xml: if isinstance(self.xml['dates']['date'], dict): return self.xml['dates']['date'].values()[0] return self.xml['dates']['date'] return None def get_publication_year(self): """Get DataCite publication year.""" if 'publicationYear' in self.xml: return self.xml['publicationYear'] return None def get_language(self): """Get DataCite language.""" if 'language' in self.xml: return self.xml['language'] return None def get_related_identifiers(self): """Get DataCite related identifiers.""" pass def get_description(self, description_type='Abstract'): """Get DataCite description.""" if 'descriptions' in self.xml: if isinstance(self.xml['descriptions']['description'], list): for description in self.xml['descriptions']['description']: if description_type in description: return description[description_type] elif isinstance(self.xml['descriptions']['description'], dict): description = self.xml['descriptions']['description'] if description_type in description: return description[description_type] elif len(description) == 1: # return the only description return description.values()[0] return None def get_rights(self): """Get DataCite rights.""" if 'titles' in self.xml: return self.xml['rights'] return None diff --git a/invenio/utils/redis.py b/invenio/utils/redis.py deleted file mode 100644 index 48f104b55..000000000 --- a/invenio/utils/redis.py +++ /dev/null @@ -1,37 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is part of Invenio. -# Copyright (C) 2015 CERN. -# -# Invenio is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# Invenio is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with Invenio; if not, write to the Free Software Foundation, Inc., -# 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. - -"""Legacy redis connector. - -.. warning:: - - DEPRECATED. -""" - -import warnings - -from invenio.utils.deprecation import RemovedInInvenio22Warning - - -def get_redis(redis_namespace='default'): - """Connect to a cache.""" - from invenio.ext.cache import cache - warnings.warn("Module invenio.utils.redis is deprecated. Please use " - "invenio.ext.cache instead.", RemovedInInvenio22Warning) - return cache diff --git a/setup.py b/setup.py index 436bc8808..b6e8e5a05 100644 --- a/setup.py +++ b/setup.py @@ -1,355 +1,354 @@ # -*- coding: utf-8 -*- # # This file is part of Invenio. # Copyright (C) 2013, 2014, 2015 CERN. # # Invenio is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License as # published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # Invenio is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Invenio; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. """Invenio is a framework for digital libraries and data repositories. Invenio enables you to run your own digital library or document repository on the web. Invenio covers all aspects of digital library management, from document ingestion, through classification, indexing and further processing, to curation, archiving, and dissemination. The flexibility and performance of Invenio make it a comprehensive solution for management of document repositories of moderate to large sizes (several millions of records). Links ----- * `website `_ * `documentation `_ * `development `_ """ import os import sys from distutils.command.build import build from setuptools import find_packages, setup from setuptools.command.install_lib import install_lib class _build(build): # noqa """Compile catalog before building the package.""" sub_commands = [('compile_catalog', None)] + build.sub_commands class _install_lib(install_lib): # noqa """Custom install_lib command.""" def run(self): """Compile catalog before running installation command.""" install_lib.run(self) self.run_command('compile_catalog') install_requires = [ "alembic>=0.6.6", "Babel>=1.3", "backports.lzma>=0.0.3", "bagit>=1.5.1", "BeautifulSoup>=3.2.1", "BeautifulSoup4>=4.3.2", "celery>=3.1.8", # Cerberus>=0.7.1 api changes and is not yet supported "Cerberus>=0.7,<0.7.1", "chardet>=2.3.0", "datacite>=0.1.0", "dictdiffer>=0.0.3", "feedparser>=5.1", "fixture>=1.5", "Flask>=0.10.1", "Flask-Admin>=1.1.0", "Flask-Assets>=0.10", "Flask-Babel>=0.9", "Flask-Breadcrumbs>=0.2", "Flask-Cache>=0.12", "Flask-Collect>=1.1.1", "Flask-Email>=1.4.4", "Flask-Gravatar>=0.4.2", "Flask-IIIF>=0.2.0", "Flask-Login>=0.2.7", "Flask-Menu>=0.2", "Flask-OAuthlib>=0.6.0,<0.7", # quick fix for issue #2158 "Flask-Principal>=0.4", "Flask-Registry>=0.2", "Flask-RESTful>=0.2.12", "Flask-Script>=2.0.5", # Development version is used, will switch to >=2.0 once released. "Flask-SQLAlchemy>=2.0", "Flask-WTF>=0.10.2", "cryptography>=0.6", "fs>=0.4", "intbitset>=2.0", "invenio-client>=0.1.0", "invenio-query-parser>=0.2", "itsdangerous>=0.24", "jellyfish>=0.3.2", "Jinja2>=2.7", "libmagic>=1.0", "lxml>=3.3", "mechanize>=0.2.5", "mistune>=0.4.1", "msgpack-python>=0.3", "MySQL-python>=1.2.5", "numpy>=1.7", "nydus>=0.10.8", # pyparsing>=2.0.2 has a new api and is not compatible yet "passlib>=1.6.2", "pyparsing>=2.0.1,<2.0.2", "python-twitter>=2.0", "pyPDF>=1.13", "pyPDF2>=1.17", "PyLD>=0.5.2", "pyStemmer>=1.3", "python-dateutil>=1.5", "python-magic>=0.4.6", "pytz>=2014.1", "rauth>=0.7.0", "raven>=5.0.0", "rdflib>=4.1.2", "redis>=2.8.0", "reportlab>=2.7,<3.2", "requests>=2.3,<2.4", "setuptools>=2.2", "six>=1.7.2", "Sphinx>=1.3", "SQLAlchemy>=1.0", "SQLAlchemy-Utils[encrypted]>=0.30.1", "unidecode>=0.04.1", "workflow>=1.2.0", "WTForms>=2.0.1", "WTForms-Alchemy>=0.13.1", "WTForms-SQLAlchemy>=0.1", "pyyaml>=3.11", ] extras_require = { "docs": [ "sphinx_rtd_theme>=0.1.7" ], "development": [ "Flask-DebugToolbar==0.9.0", "watchdog==0.8.3", ], "dropbox": [ "dropbox>=2.1.0" ], "elasticsearch": [ "pyelasticsearch>=0.6.1" ], "googledrive": [ "google-api-python-client>=1.2", "apiclient>=1.0.0", "oauth2client>=1.4.0", "urllib3>=1.8.3" ], "img": [ "qrcode>=5.1", "Pillow>=2.7.0" ], "mongo": [ "pymongo>=3.0" ], "misc": [ # was requirements-extras "gnuplot-py==1.8", "flake8>=2.0.0", # extra=kwalitee? "pychecker==0.8.19", # extra=kwalitee? "pylint>=1.4.0", # extra=kwalitee? "nosexcover>=1.0.0", # test? "python-onedrive>=15.0.0", # extra=cloud? "python-openid>=2.2.0", # extra=sso? ], "mixer": [ "mixer>=5.1.0", ], "sso": [ "Flask-SSO>=0.2" ], "postgresql": [ "psycopg2>=2.5", ], # Alternative XML parser # # For pyRXP, the version on PyPI many not be the right one. # # $ pip install # > https://www.reportlab.com/ftp/pyRXP-1.16-daily-unix.tar.gz#egg=pyRXP # "pyrxp": [ # Any other versions are not supported. "pyRXP==1.16-daily-unix" ], "rabbitmq": [ "amqp>=1.4.5", ], "github": [ "github3.py>=0.9" ], } extras_require["docs"] += extras_require["elasticsearch"] extras_require["docs"] += extras_require["img"] extras_require["docs"] += extras_require["mongo"] extras_require["docs"] += extras_require["sso"] extras_require["docs"] += extras_require["github"] # FIXME extras_require["docs"] += extras_require["dropbox"] # FIXME extras_require["docs"] += extras_require["googledrive"] tests_require = [ "httpretty>=0.8.4", "Flask-Testing>=0.4.1", "mock>=1.0.0", "nose>=1.3.0", "selenium>=2.45.0", "unittest2>=0.5", ] setup_requires = [ 'Babel>=1.3', ] # Add `tests` dependencies to `extras_require` so that developers # could install test dependencies also with pip: extras_require["tests"] = tests_require # Compatibility with Python 2.6 if sys.version_info < (2, 7): install_requires += [ "argparse>=1.3.0", "importlib>=1.0.0" ] # Get the version string. Cannot be done with import! g = {} with open(os.path.join("invenio", "version.py"), "rt") as fp: exec(fp.read(), g) version = g["__version__"] packages = find_packages(exclude=['docs']) packages.append('invenio_docs') setup( name='invenio', version=version, url='https://github.com/inveniosoftware/invenio', license='GPLv2', author='CERN', author_email='info@invenio-software.org', description='Invenio digital library framework', long_description=__doc__, packages=packages, package_dir={'invenio_docs': 'docs'}, include_package_data=True, zip_safe=False, platforms='any', entry_points={ 'console_scripts': [ 'inveniomanage = invenio.base.manage:main', 'plotextractor = invenio.utils.scripts.plotextractor:main', # Legacy 'alertengine = invenio.legacy.webalert.scripts.alertengine:main', 'batchuploader = invenio.legacy.bibupload.scripts.batchuploader', 'bibcheck = invenio.legacy.bibcheck.scripts.bibcheck:main', 'bibcircd = invenio.legacy.bibcirculation.scripts.bibcircd:main', 'bibauthorid = ' ' invenio.legacy.bibauthorid.scripts.bibauthorid:main', 'bibcatalog = invenio.legacy.bibcatalog.scripts.bibcatalog:main', 'bibclassify = invenio.modules.classifier.scripts.classifier:main', 'bibconvert = invenio.legacy.bibconvert.scripts.bibconvert:main', 'bibdocfile = invenio.legacy.bibdocfile.scripts.bibdocfile:main', 'bibedit = invenio.legacy.bibedit.scripts.bibedit:main', 'bibencode = invenio.modules.encoder.scripts.encoder:main', 'bibexport = invenio.legacy.bibexport.scripts.bibexport:main', 'bibindex = invenio.legacy.bibindex.scripts.bibindex:main', 'bibmatch = invenio.legacy.bibmatch.scripts.bibmatch:main', 'bibrank = invenio.legacy.bibrank.scripts.bibrank:main', 'bibrankgkb = invenio.legacy.bibrank.scripts.bibrankgkb:main', - 'bibreformat = invenio.legacy.bibformat.scripts.bibreformat:main', 'bibsort = invenio.legacy.bibsort.scripts.bibsort:main', 'bibsched = invenio.legacy.bibsched.scripts.bibsched:main', 'bibstat = invenio.legacy.bibindex.scripts.bibstat:main', 'bibtaskex = invenio.legacy.bibsched.scripts.bibtaskex:main', 'bibtasklet = invenio.legacy.bibsched.scripts.bibtasklet:main', 'bibtex = invenio.modules.sequencegenerator.scripts.bibtex:main', 'bibupload = invenio.legacy.bibupload.scripts.bibupload:main', 'convert_journals = ' ' invenio.legacy.docextract.scripts.convert_journals:main', 'dbexec = invenio.legacy.miscutil.scripts.dbexec:main', 'dbdump = invenio.legacy.miscutil.scripts.dbdump:main', 'docextract = invenio.legacy.docextract.scripts.docextract:main', 'elmsubmit = invenio.legacy.elmsubmit.scripts.elmsubmit:main', 'gotoadmin = invenio.modules.redirector.scripts.redirector:main', 'hepdataharvest = ' ' invenio.utils.hepdata.scripts.hepdataharvest:main', 'inveniogc = invenio.legacy.websession.scripts.inveniogc:main', 'inveniounoconv = ' ' invenio.legacy.websubmit.scripts.inveniounoconv:main', 'oaiharvest = invenio.legacy.oaiharvest.scripts.oaiharvest:main', 'oairepositoryupdater = ' ' invenio.legacy.oairepository.scripts.oairepositoryupdater:main', 'arxiv-pdf-checker = invenio.legacy.pdfchecker:main', 'refextract = invenio.legacy.refextract.scripts.refextract:main', 'textmarc2xmlmarc = ' ' invenio.legacy.bibrecord.scripts.textmarc2xmlmarc:main', 'webaccessadmin = ' ' invenio.modules.access.scripts.webaccessadmin:main', 'webauthorprofile = ' ' invenio.legacy.webauthorprofile.scripts.webauthorprofile:main', 'webmessageadmin = ' ' invenio.legacy.webmessage.scripts.webmessageadmin:main', 'webstatadmin = invenio.legacy.webstat.scripts.webstatadmin:main', 'websubmitadmin = ' ' invenio.legacy.websubmit.scripts.websubmitadmin:main', 'xmlmarc2textmarc = ' ' invenio.legacy.bibrecord.scripts.xmlmarc2textmarc:main', 'xmlmarclint = invenio.legacy.bibrecord.scripts.xmlmarclint:main', ], "distutils.commands": [ "inveniomanage = invenio.base.setuptools:InvenioManageCommand", ] }, setup_requires=setup_requires, install_requires=install_requires, extras_require=extras_require, classifiers=[ 'Development Status :: 4 - Beta', 'Environment :: Web Environment', 'Intended Audience :: Developers', 'License :: OSI Approved :: GNU General Public License v2' ' or later (GPLv2+)', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Topic :: Internet :: WWW/HTTP :: Dynamic Content', ], test_suite='invenio.testsuite.suite', tests_require=tests_require, cmdclass={ 'build': _build, 'install_lib': _install_lib, }, )